[ankurgpt-71b9] Details ============================ Generated On: 2024-11-20 15:34:51 UTC TML Solution DAG Parameters' Details: User Chosen Parametets ---------------------------- STEP 1: Get TML Core Params: `tml_system_step_1_getparams_dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - solutionname - ankurgpt-71b9 * - solutiontitle - My Solution Title * - solutiondescription - This is an awesome real-time solution built by TSS * - brokerhost - 127.0.0.1 * - brokerport - 9092 * - cloudusername - None * - ingestdatamethod - MQTT STEP 2: Create Kafka Topics: `tml_system_step_2_kafka_createtopic_dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - companyname - Otics * - myname - Sebastian * - myemail - Sebastian.Maurice * - mylocation - Toronto * - replication - 1 * - numpartitions - 1 * - enabletls - 1 * - microserviceid - * - raw_data_topic - cisco-network-mainstream * - preprocess_data_topic - cisco-network-preprocess * - ml_data_topic - ml-data * - prediction_data_topic - prediction-data STEP 3: `Produce to Kafka Topics `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - PRODUCETYPE - MQTT * - TOPIC - cisco-network-mainstream * - PORT - 38777 * - IDENTIFIER - MQTT Subscription Topic: tml/cybersecurity * - HTTPADDR - https:// * - FROMHOST - Rog,127.0.1.1 * - TOHOST - 127.0.1.1 * - CLIENTPORT - 8883 * - TSS_CLIENTPORT - 8883 * - TML_CLIENTPORT - 8883 STEP 4: Preprocesing Data: `tml-system-step-4-kafka-preprocess-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - raw_data_topic - cisco-network-mainstream * - preprocess_data_topic - cisco-network-preprocess * - preprocessconditions - * - delay - 70 * - array - 0 * - saveasarray - 1 * - topicid - -999 * - rawdataoutput - 1 * - asynctimeout - 120 * - timedelay - 0 * - preprocesstypes - min,max,trend,anomprob,variance,avg * - pathtotmlattrs - --pathtotmlattrs-- * - identifier - Detect potential cyber attacks and monitor network * - jsoncriteria - uid=hostName,filter:allrecords~ subtopics=hostName,hostName,hostName~ values=inboundpackets,outboundpackets,pingStatus~ identifiers=inboundpackets,outboundpackets,pingStatus~ datetime=lastUpdated~ msgid=~ latlong= STEP 4b: Preprocesing Data: `tml-system-step-4b-kafka-preprocess-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - raw_data_topic - --raw_data_topic2-- * - preprocess_data_topic - --preprocess_data_topic2-- * - preprocessconditions - --preprocessconditions2-- * - delay - --delay2-- * - array - --array2-- * - saveasarray - --saveasarray2-- * - topicid - --topicid2-- * - rawdataoutput - --rawdataoutput2-- * - asynctimeout - --asynctimeout2-- * - timedelay - --timedelay2-- * - preprocesstypes - --preprocesstypes2-- * - pathtotmlattrs - --pathtotmlattrs2-- * - identifier - --identifier2-- * - jsoncriteria - --jsoncriteria2-- STEP 5: Entity Based Machine Learning : `tml-system-step-5-kafka-machine-learning-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - preprocess_data_topic - cisco-network-preprocess * - ml_data_topic - ml-data * - modelruns - --modelruns-- * - offset - -1 * - islogistic - --islogistic-- * - networktimeout - --networktimeout-- * - modelsearchtuner - --modelsearchtuner-- * - processlogic - --processlogic-- * - dependentvariable - --dependentvariable-- * - independentvariables - --independentvariables-- * - rollbackoffsets - --rollbackoffsets-- * - topicid - -999 * - consumefrom - cisco-network-preprocess * - fullpathtotrainingdata - --fullpathtotrainingdata-- * - transformtype - --transformtype-- * - sendcoefto - --sendcoefto-- * - coeftoprocess - --coeftoprocess-- * - coefsubtopicnames - --coefsubtopicnames-- STEP 6: Entity Based Predictions: `tml-system-step-6-kafka-predictions-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - preprocess_data_topic - cisco-network-preprocess * - ml_prediction_topic - --ml_prediction_topic-- * - streamstojoin - --streamstojoin-- * - inputdata - --inputdata-- * - consumefrom - --consumefrom2-- * - offset - -1 * - delay - 70 * - usedeploy - --usedeploy-- * - networktimeout - --networktimeout-- * - maxrows - --maxrows-- * - topicid - -999 * - pathtoalgos - --pathtoalgos-- STEP 7: Real-Time Visualization: `tml-system-step-7-kafka-visualization-dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - vipervizport - 9005 * - topic - cisco-network-preprocess,cisco-network-privategpt * - dashboardhtml - tml-cisco-network-privategpt-monitor.html * - secure - 1 * - offset - -1 * - append - 0 * - chip - amd64 * - rollbackoffset - 400 STEP 8: `tml_system_step_8_deploy_solution_to_docker_dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - Docker Container - pokhrelankur8@gmail.com/ankurgpt-71b9-amd64 (https://hub.docker.com/r/pokhrelankur8@gmail.com/ankurgpt-71b9-amd64) * - Docker Run Command - docker run -d -p 56539:56539 -p 41161:41161 -p 56181:56181 -p 8883:8883 \-\-env TSS=0 \-\-env SOLUTIONNAME=ankurgpt-71b9 \-\-env SOLUTIONDAG=solution_preprocessing_ai_mqtt_dag-ankurgpt-71b9 \-\-env GITUSERNAME=AnkurPokhrel8 \-\-env GITREPOURL=https://github.com/AnkurPokhrel8/raspberryCYT180 \-\-env SOLUTIONEXTERNALPORT=56539 -v /var/run/docker.sock:/var/run/docker.sock:z \-\-env CHIP=amd64 \-\-env SOLUTIONAIRFLOWPORT=41161 \-\-env SOLUTIONVIPERVIZPORT=56181 \-\-env DOCKERUSERNAME='pokhrelankur8@gmail.com' \-\-env CLIENTPORT=8883 \-\-env EXTERNALPORT=38777 \-\-env KAFKACLOUDUSERNAME='' \-\-env VIPERVIZPORT=9005 \-\-env MQTTUSERNAME='GyanRaj8' \-\-env AIRFLOWPORT=9000 \-\-env GITPASSWORD='' \-\-env KAFKACLOUDPASSWORD='' \-\-env MQTTPASSWORD='' \-\-env READTHEDOCS='' pokhrelankur8@gmail.com/ankurgpt-71b9-amd64 STEP 9: `tml_system_step_9_privategpt_qdrant_dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - PrivateGPT Container - maadsdocker/tml-privategpt-no-gpu-amd64 * - PrivateGPT Run Command - docker run -d -p 8001:8001 --net=host --gpus all --env PORT=8001 --env GPU=1 --env COLLECTION=tml --env WEB_CONCURRENCY=1 --env CUDA_VISIBLE_DEVICES=0 maadsdocker/tml-privategpt-no-gpu-amd64 * - Qdrant Container - qdrant/qdrant * - Qdrant Run Command - docker run -d -p 6333:6333 -v $(pwd)/qdrant_storage:/qdrant/storage:z qdrant/qdrant * - Consumefrom - cisco-network-preprocess * - pgpt_data_topic - cisco-network-privategpt * - offset - -1 * - rollbackoffset - 400 * - topicid - -999 * - enabletls - 1 * - partition - -1 * - prompt - Do any of the values of the inbound or outbound packets look abnormal? * - context - These data are anomaly probabilities of suspicious data traffic. A high probability of over 0.80 is likely suspicious. * - jsonkeytogather - hyperprediction * - keyattribute - outboundpackets,inboundpackets * - keyprocesstype - anomprob * - vectordbcollectionname - tml * - concurrency - 1 * - CUDA_VISIBLE_DEVICES - 0 * - pgpthost - http://127.0.0.1 * - pgptport - 8001 * - hyperbatch - 0 STEP 10: `tml_system_step_10_documentation_dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - Solution Documentation URL - https://ankurgpt-71b9.readthedocs.io