o 4jsi@sddlZddlmZddlZddlmZddlmZddlm Z ddl m Z ddl m Z dd lmZdd lmZe Ze ZeZeZdd lmZddlZddlZdd lmZdd lmZddlZddlZddZeZ GdddZ!dS)N) HTTPException) HashGenerator) UserJourney)MicrolearningDurationEstimator)CourseOutlineParser)ContentReviewer) Synthesizer)LiveWhisperSTTTool)Path)gTTS) AudioSegmentcs~tjtjtfdd}|}|r9|t_tjtj|d}tj|r-|t_ n|t_ t d|dSt ddS)z!Configure pydub with ffmpeg pathscsvtjd}tj|r|StddD]$}tjjgdg|R}tjtj|d}tj|r8|SqdS)Nz ffmpeg.exez..)ospathjoinexistsrangeabspath) ffmpeg_pathi parent_dir current_dir/home/azureuser/microlearn/backend/user_journey_with_openai/agentic_workflow/src/user_journey_service/services/content_creation_service.py find_ffmpegs  z$configure_pydub..find_ffmpegz ffprobe.exeu✓ Pydub configured with: Tu=⚠ Warning: ffmpeg.exe not found. Audio conversion may fail.F) rrdirnamer__file__r converterrrffprobeprint)rr ffprobe_pathrrrconfigure_pydubs  r$c@seZdZddZddZddZddZd d Zd d Zd dZ ddZ ddZ ddZ ddZ ddZddZddZddZdd Zd!d"Zd#d$Zd%d&Zd'd(Zd)d*Zd+d,Zd-d*Zd.d/Zd0d1Zd2d3Zd4d5Zd6d7Zd8S)9ContentCreationServicecCsnt|_||_t||_d|jd|_d|jd|_d|jd|_d|jd|_ d|jd|_ dS)Nz research/.mdzoutput/zoutput1/zoutput2/zparsed_course_content/z.json) r crew_instance input_datargenerate_input_hash input_hash research_file output_file output_file_1 output_file_2json_output_path)selfr(rrr__init__Fs zContentCreationService.__init__cCsX|dd}td|jd|d|d}td|jd|d|d}|o+|S)z9Check if content and questions already exist for a lesson._updated_content//module_lessonr& question/)replacer r*r)r0 module_idx lesson_number lesson_safeupdated_content_file question_filerrr_has_updated_content_questionsQs z5ContentCreationService._has_updated_content_questionsc Cstd|dd}td|jd|d|d}td|jd|d|d }td|jd|d|d }td|jd|d|d }|oU|oU|oU|}td ||S) z'Check if audio files exist for a lessonz.Inside audio files availability check functionr2r3audio/lessons/r5r6.wavaudio/questions/z/1.wavz/2.wavz/3.wavzThe output is : )r"r8r r*r) r0r9r:r; audio_contentaudio_question1audio_question2audio_question3resultrrr _has_audio_content_and_questions[s z7ContentCreationService._has_audio_content_and_questionsc Csi}td|jd|d}td|td|jd|d}t|d}|D]-}t|||}tdtd ttd }||d } |||<|j } | j | d q*d |i} |j } | j | d dS)N audio/stage//stager@z$The audio content is available at : rA/*.wavu/🎤 Press Enter when you're ready to answer...u 📢 Listening to your answer...zEnter your answer: )questionanswerinputsusers_response) r r*r"sortedglobwav_to_text_whisperinputstrr'evaluator_crewkickoffassessment_crew) r0idxall_qns_answerrBaudio_question wav_fileswav_filerLrMrOcrewrrr_run_evaluationps$     z&ContentCreationService._run_evaluationc Cs|dd}td|jd|d|d}|r!td|n@tjtjd|jdd d td |jd|d|d }t |d dd }| }Wdn1sVwY| ||td|jd|d|d}t | d} | stj|d d td|||dSdS)zCreate audio files for a lessonr2r3r?r5r6.mp3z/Audio for the content is already available at: rJTexist_okr4r&rutf-8encodingNrA*.mp3Question audio is not available)r8r r*rr"rmakedirsrropenread text_to_wavrQrR_create_audio_question) r0r9r:r;rB content_filef text_contentaudio_question_dirr\rrr_create_audio_filess"    z*ContentCreationService._create_audio_filesc Cs|dd}td|jd|d|d}tjtjd|jddd td |jd|d|d }t|d d d }|}Wdn1sJwY| ||td|jd|d|d}tj|dd t d| ||dS)z>Create audio files for a lesson without checking if they existr2r3r?r5r6r`rJTrar4r&rcrdreNrArh) r8r r*rrirrrjrkrlr"rm) r0r9r:r;rBrnrorprqrrr!_create_audio_files_without_checks    z8ContentCreationService._create_audio_files_without_checkcCstd|jd|d}|rtd|n=tjtjd|jdddtd|jd|d }t|d d d  }| }Wdn1sJwY| ||td |jd|d}t | d}|stjtjd |jd|dddtd| |dSdS)NrHrIr@z/Audio for the content is already available at :rJTrar4r&rcrdrerArKzquestion audio is not available)r r*rr"rrirrrjrkrlrQrRrm)r0rYrBrnrorpr[r\rrr_create_audio_files_04_01_2026s    &z5ContentCreationService._create_audio_files_04_01_2026c Cs|dd}td|jd|d|d}|rt|ddd  }|}Wd n1s/wY||}td t|d |d |t |ddD]1\}} td|d|d |d| d dd td|jd|d|d|d } | | | qNd Std|d S)z6Create audio files for questions for a specific lessonr2r3r7r5r6r&rcrdreNzFound z* questions to convert to audio for Module z , Lesson rstartu 🧠 Asking Question z for Module : dz... rArJr@zQuestion file not found: ) r8r r*rrjrkparse_questionsr"len enumeraterl) r0r9r:r;r=ro questionsparsed_questionsindexrLr[rrrrms   *$z-ContentCreationService._create_audio_questioncCs|d}g}g}|D]8}|}|ds|dr7|r)|d|g}tdd|}tdd|}|rC|dsC||q |rP|d|dd |D}|smd }t||tjtj B}d d |D}|d d S)z"Parse questions from markdown text z### Q z ^### Q\d+:\s*z^### Q\d+:.*?\s*z###cSs g|] }|rt|dkr|qS) )rz.0qrrr s z:ContentCreationService.parse_questions..z#### Q\d+:?\s*(.*?)(?=\n### Q\d+:|$)cSg|] }|r|qSrstriprrrrrN ) splitr startswithappendrresubfindallDOTALL IGNORECASE)r0md_textlinesr|current_questionlinepatternrrrrys,   z&ContentCreationService.parse_questionscCs|sdSt|}gd}|D]}tj|d|tjtjBd}qdddddddd}|D]\}}tj|||tjtjBd}q,tdd|}g}|dD]'}|}|rs|rn|d  rnt |d krn|d  |d d }| |qLd |}td d |}|S)zz Intelligently clean text for audio conversion. Handles various markdown and formatting patterns. r) z ^#{1,6}\s+z \*\*|\*|__|_z ```.*?```z`[^`]+`z!?\[.*?\]\(.*?\)z&[a-z]+;z ^\s*[-*+]\s+z ^\s*\d+\.\s+z^>\s+z\|-+\|z ^\|.*?\|$flagsz\1. )z^#{1,3}\s+(.*?)\s*$z^\s*[-*+]\s+(.*?)\s*$z^\s*\d+\.\s+(.*?)\s*$zA^(Introduction|Conclusion|Summary|Note|Tip|Warning|Important):\s*z"^(Step \d+|Phase \d+|Part \d+):\s*z^(Q\d+|Question \d+)[:.]?\s*z'^(Easy|Moderate|Difficult|Hard)[-:]?\s*z [.!?]{2,}r2rrrNrz\s+)rUrr MULTILINEritemsrrrislowerrzupperrr)r0textpatterns_to_remover replacements replacementrrrrrintelligent_text_cleaners8  z/ContentCreationService.intelligent_text_cleanerc Cs|sdS|d}g}|D]}|}|sq |dr*|d|ddq |dr>|d|d dq |d rR|d |d dq |d rd||ddq |dss|dss|dr|dd}tdd|}tdd|}|d|q td|rtdd|}tdd|}|d|q |}tdd|}tdd|}tdd|}tdd|}|r||q g}t|D]\}}|||ds|dr|dqd|S)z Optimal markdown to speech converter - simpler but effective. Preserves semantic meaning without over-engineering. rr# z Main topic: rN## Section: ### Subsection: r#### z- z* z+ z[\*_]{1,2}(.*?)[\*_]{1,2}\1z `([^`]+)`zcode \1• ^\d+\.\s+ \*\*(.*?)\*\* \*(.*?)\*\[([^\]]+)\]\([^)]+\))z Main topic:Section:z Subsection:) rrrrrrmatchr{r) r0rr speech_linesritem clean_linerFrrrroptimal_markdown_to_speechGsT          z1ContentCreationService.optimal_markdown_to_speechc Cs|sdSt|}|d}g}|D]d}|}|sq|dr1|dd}|d|q|drG|dd}|d |q|d r]|d d}|d |q|d rs|dd}|d|q|dr|dd}|d|q|dr|dd}|d|qtd|rtdd|}||}|d|qtd|rtd|}|r| d}| d}||}|d|d|qtd|rtdd|}||}d| vsd|vr |d|q|d |q|d!r0|dd}||}|d"|qtd#|r=|d$qd%|vrit d&|ritd'|shd(d)|d%D}|rh|d*d+ |q||}|rv||q| d |S),z Convert markdown to natural speech while preserving semantic meaning. Handles: # Headings, **bold**, *italic*, - bullets, 1. numbered lists, `code`, [links](url) rrz###### Nrz##### rzMinor section: rrrrrzTopic: rrz Chapter: rrzTitle: z ^[-*+]\s+rrz^(\d+)\.\s+(.*)Point rwz ^\[[ xX]\]\s+z[x]z[X]z Completed: zTo do: z> zQuote: z ^[-*_]{3,}$z---|z \w.*\|.*\wz^\|?[-:| ]+\|?$cSrrr)rcellrrrrrz=ContentCreationService.markdown_to_speech..z Table row: z; )rUrrrrrrr_remove_inline_formattinggrouplowersearchr_add_speech_pauses) r0rr result_linesrcontentrnumbercellsrrrmarkdown_to_speechs|                     z)ContentCreationService.markdown_to_speechcCs|sdS|}dd}dd}dd}dd }td ||}td ||}td ||}td ||}td||}td||}tdd|}tdd|}|dd}|dd}|dd}|dd}|S)zV Remove inline markdown formatting while preserving semantic meaning. rcS|d}|SNrrrrrrr replace_bold zFContentCreationService._remove_inline_formatting..replace_boldcSrrrrrrrreplace_italicrzHContentCreationService._remove_inline_formatting..replace_italiccS|d}d|S)Nrzcode: rrrrrreplace_inline_code  zMContentCreationService._remove_inline_formatting..replace_inline_codecSr)Nrzstrikethrough: rrrrrreplace_strikethrough rzOContentCreationService._remove_inline_formatting..replace_strikethroughrrz __(.*?)__z_(.*?)_z`(.*?)`z ~~(.*?)~~rrz!\[([^\]]*)\]\([^)]+\)z Image: \1z rz&andz<z less thanz>z greater than)rrr8r)r0roriginalrrrrrrrrs(    z0ContentCreationService._remove_inline_formattingcCs|d}g}t|D]_\}}|}|sq |dr&|||dq |drG|||t|dkrF||ddsF|dq |||t|dkrj||drj||ddsj|dq d|S)zK Add natural pauses for speech based on content structure. r)Title:Chapter:zTopic:rr)rrz Completed:zTo do:r)rr)rrrr)rr{rrrrzr)r0rrrFrrrrrr#s&      "  2  z)ContentCreationService._add_speech_pausesc Csttd|z]t|}||}|stdWdSddl}d|jddd}td|t|d d d }||td t| d d}t j |rbt ||td||WSWdSt y}zItd|ddl}|t|d} t| ddd} | d|d|ddWdn1swYtd| WYd}~dSd}~ww)z3Convert text to WAV audio file - SIMPLIFIED VERSIONConverting text to audio: ,Warning: Empty text, skipping audio creationNr temp_audio_r`Creating MP3 with gTTS: enFrlangslow✓ gTTS conversion completer@✓ Audio saved as ❌ Error in text_to_wav: .error.txtwrdreError:  Text: Saved error details to: )r"rUrruuiduuid4hexr saver8rrr Exception traceback print_excrjwrite) r0rr,rtemp_mp3tts mp3_outputer error_filerorrrtext_to_wav_oldCs@       z&ContentCreationService.text_to_wav_oldc Cstd|zxt|}||}|stdWdSddl}d|jddd}td|t|d d d }||td t |}|j j d d dtd|j t j |rvddl}||t|td|t|WStd|WdSty}z[td|ddl}|t|d} z/t| ddd} | d|d|ddWdn1swYtd| WntdYWYd}~dSWYd}~dSd}~ww)z.Convert text to WAV audio file - FIXED VERSIONrrNrrrr`rrFrrTparentsrbu ✓ Directory created/verified: ru❌ Temp file not found: rrrrdrerrrrzCould not save error file)r"rUrrrrrr rr parentmkdirrrrshutilmoverrrrjr) r0rr,rrrrrrrrorrrrlzsP        z"ContentCreationService.text_to_wavcCs6tdtd}|t|}td|d|dS)Nzinside wave to text conversionbaseu🎤 Transcription:r)r"whisper load_model transcriberU)r0r]modelrFrrrrSs  z*ContentCreationService.wav_to_text_whisperc Csztj|jsdddWStdt|j}|}td|dd|d}dddddd }t|d d d D]\}}|d }t dd| }td|d|t|dd d D]f\} } | d} t d| } | rr| d } n|d| } t dd|  }|dd 7<td| d|| || }|d|d7<|d|d7<|d|d7<|d|d 7<q\q9d!d"|j|d#|dd$|dd%|dd&d'WSty }zdd(l}|}td)|td*|td+d,|d-d(}~ww).z6Creates audio files for existing content and questionsfailureUser journey is not created.statusmessagez"Starting audio creation process...Course Title: course_titlerr) total_lessonscontent_audio_createdquestion_audio_createdskipped_contentskipped_questionsmodulesrru module_title^Module \d+:\s*rProcessing Module rwlessons lesson_titleLesson (\d+\.\d+)r2^Lesson \d+\.\d+:\s*r Processing Lesson rcontent_createdr questions_createdr content_skippedr questions_skippedsuccesszAudio creation completed.zCreated z content audio files and z question audio files for z lessons.)rrhashidstatssummaryNzERROR in run_audio_creation: z Traceback: rz$Unexpected error in audio creation:  status_codedetail)rrrr,r"r parse_contentr{rrrrr_create_lesson_audio_filesr*rr format_excr)r0parser parsed_resultr audio_statsr9moduler  module_name lesson_idxlessonlesson_title_fulllesson_number_matchr: lesson_namelesson_audio_statsrr error_detailsrrrrun_audio_creations^      "z)ContentCreationService.run_audio_creationc Csddddd}|dd}td|jd|d|d}td |jd|d|d }|r|std z/t|d d d }|}Wdn1sNwY||||dd7<tdWn-ty} z td| WYd} ~ nd} ~ wwtd|dd7<ntd|td|jd|d|d} td|jd|d|d } | r_| rt | dng} | s| j ddd| rtdt | d|dt | 7<|Std zXt| d d d }|} Wdn 1swY| | }d}t|dd!D]\}}| |d}|s/||||d7}q|d"|7<td#|d$W|Sty^} ztd%| WYd} ~ |Sd} ~ wwtd&| |S)'z9Create audio files for a specific lesson and return statsr)rrrrr2r3r?r5r6r`r4r&z Creating content audio...rcrdreNrru ✓ Content audio createdu& ✗ Error creating content audio: z Content audio already existsrz% Warning: Content file not found: rArJr7rgTrz z# question audio files already existrz$ Creating question audio files...ruru ✓ Created z question audio filesu' ✗ Error creating question audio: z& Warning: Question file not found: )r8r r*rr"rjrkrlrrQrRrrzryr{)r0r9r:rr;content_audio_pathrnrorprquestion_audio_dirr=existing_audior|r} created_countr~rLaudio_question_pathrrrr!st         z1ContentCreationService._create_lesson_audio_filescCszTtj|jsdddWStj|jrStdt|j}|}tjtj|j dd| |j |td|dd |d}d d d d d d }t |d d dD]\}}|d}|d}|d} |d} td|d|t dd|} t |dd dD]\} } | d}| d}| d}| d}t d|}|r|d }n|d| }t dd|}|dd 7<td|d|t d|}|rt|}|d }|d!}nd"}d#}d$|jd%|d&|dd'd(}d)|jd%|d&|dd'd(}|||r3td*|j|| || | ||||||||d+ }|d,d 7<|d-d 7<qtd.|d/d 7<|d0d 7<qqUd1d2|j|d3d4WSWd8Styk}z td5d6|d7d8}~ww)9>Creates content for each module and lesson (NO audio creation)rrrThe user journey is createdTrarrrr)rrcontent_updatedrquestions_updatedr rrur focusoutcomedurationrrwrrrrtopicsrr2rrr(\d+)x` ( content/r5r6r3r&r7) Creating new content and questions... r9r' module_focusmodule_outcomemodule_durationr:r,lesson_durationlesson_outcome lesson_topicsrword_count_lowerword_count_upperrr' Content and questions already existr7r8rAContent and questions created (audio will be created separately).-Call /run-audiocreation to create audio filesrrrr next_steprError parsing result: rN)rrrr,r"rr rirr/export_to_jsonr{rrrrrintr*r8_should_create_content_create_new_lessonrr)r0r#r$r content_statsr9r&r rErFrGr'r(r)r*rHrIrJr+r:r,duration_matchminutes lower_range upper_rangernr=rFrrrrrun_content_creation_23012026Ls        "";`z4ContentCreationService.run_content_creation_23012026c Ct|dd}tjd|jd|d|d}tjd|jd|d|d}|r2|r0|s2dS|s8dSd S z$Check if content needs to be createdr2r3rBr5r6r&r7TFr8rrrr*_is_first_iterationr0r9r:r;content_existsquestion_existsrrrrU ""z-ContentCreationService._should_create_contentcCspztj|jsdddWStj|jrtdt|j}|}tjtj|j dd| |j |td|dd |d}d d d d d d d d }t |d d dD]<\}}|d}|d}|d} |d} td|d|t dd|} t |dd dD] \} } | d}| d}| d}| d}t d|}|r|d }n|d| }t dd|}|dd 7<td|d|t d|}|rt|}|d }|d!}nd"}d#}|dd$}d%|jd&|d'|d(}d)|jd&|d'|d(}|||rtd*|j|| || | ||||||||d+ }|d,d 7<|d-d 7<tj|rY||}|rY|d.d 7<|d/rY|d/d 7<tj|r~||}|r~|d.d 7<|d/r~|d/d 7<qtd0|d1d 7<|d2d 7<qqWd3d4|j|d5d6WSWd:Sty}z td7d8|d9d:}~ww);r5rrrr6Trarrrr)rrr7rr8word_count_removedspecial_chars_cleanedr rrur r9r:r;rrwrrrrr<rr2rrrr=r>r?r@rAr3rBr5r6r&r7rCrDrrrerfrMr7r8rrNrOrPrrRrN)rrrr,r"rr rirr/rSr{rrrrrrTr8r*rUrV_clean_content_filegetrr)r0r#r$rrWr9r&r rErFrGr'r(r)r*rHrIrJr+r:r,rXrYrZr[r;rnr=rFcleanedrrrrrun_content_creations             Krz+ContentCreationService.run_content_creationc Cr]r^r_rarrrrU@rdcCs&td|jj|jjdkrdSdS)NzInside feedback test : zFirst iterationTF)r"r(feedback)r0rrrr`Os z*ContentCreationService._is_first_iterationc Cs(zt|ddd }|}Wdn1swY|}d}d}gd}|D]}tj|d|tjtjBd} | |kr?d }| }q(d dl} t| j} | d g} |D]%} | dks`| d ks`| d krf| | qR| | vrp| | qR| dd }qRd | }tdd|}tdd|}tdd|}tdd|}tdd|}dd| dD}d |}| dd}||}||krt|ddd }||Wdn1swYtdtj|d |dWSddiWSty}ztd|d |dt|d!WYd}~Sd}~ww)"zAClean content file by removing word counts and special charactersrcrdreNF)>\n*(?:Word\s*Count|word\s*count|WORD\s*COUNT)\s*:?\s*\d+\s*\n*/\n*CONTENT_(?:REJECTED|TOO_SHORT).*?\n*(?=\n|$)D\n*###?\s*(?:Validation|Word Count|Feedback).*?(?=\n###|\n##|\n#|\Z)z+\n*(?:Reading time|Estimated reading).*?\n*z'\n*(?:Retry|Attempt|Regeneration).*?\n*\n\s*Total\s*$\n\s*Total\s*\n*$rrTr—–•°´`‘’"“”  rr[ \t]+ \n\s*\n\s*\n+ z\*\*\s+z**z\s+\*\*z#\s+rcSsg|]}|qSr)rstrip)rrrrrrsz>ContentCreationService._clean_content_file..ru ✓ Cleaned: )rirfriu ✗ Error cleaning rw)rierror)rjrkrrrrstringset printableupdaterrrr8_remove_trailing_metadatarr"rrbasenamerrU)r0 file_pathrororiginal_contentrirfword_count_patternsr new_contentrz allowed_charscleaned_contentcharrrrrrrgUsb             z*ContentCreationService._clean_content_filecCs^|j}td|}|rt|}nd}|dd}d|jd|d|d}d |jd|d|d}| |||||d |||| | d d }td |td| d| td|jj d|jj | | t |||||||| | || d| dd td||j j|d}|j|d||t|ddd }|}Wdn1swYddd| D}td|t|| ||| d| }||}|j j|d}||d<d |d!<|j|d||d"|jd|d|d}tjtj|d#d$t|d%dd }||Wdn 1s#wY||d&S)'z4Create content for a new lesson WITH ENHANCED PROMPTr=r2r3rBr5r6r&r7)rnamer9r:r;)rtitler; objectivesr<)CourseModuleLessonz*The user journey for content creation is: z(The word count for content creation is: z to z.The level and motive for content creation is: z and a ## CRITICAL FORMATTING RULES - MUST FOLLOW: 1. **NO WORD COUNT REFERENCES:** - DO NOT include "Word Count:", "word count:", "Total words:" or any word count information - DO NOT mention word count validation in the content - DO NOT add word count numbers anywhere - DO NOT add standalone word "Total" at the end of content 2. **NO VALIDATION COMMENTS:** - DO NOT include "CONTENT_REJECTED", "CONTENT_TOO_SHORT", or validation feedback - DO NOT mention validation or approval processes - Only output the actual lesson content 3. **CLEAN SPECIAL CHARACTERS:** - Use standard ASCII characters only - Avoid Unicode special characters that may cause display issues - Use standard punctuation: , . ! ? : ; " ' 4. **PROPER MARKDOWN FORMATTING:** - Use consistent heading levels: # for main title, ## for sections, ### for subsections - Use proper line breaks and spacing - Ensure all markdown syntax is correctly formatted 5. **OUTPUT FORMAT:** - Only output the lesson content in clean markdown - No extra comments, notes, or explanations - No word count information - No validation messages ) r module_numberr'rErFr:rrHlesson_objectivesrJ user_journey word_countstrict_formattingzCreating content file: )r,rNrcrdreNz, and cSsg|]}d|dqS)'r)rtopicrrrrsz=ContentCreationService._create_new_lesson..zThe topic sections are: ra ## IMPORTANT FORMATTING RULES: - DO NOT include word count or validation comments - Use clean, standard characters only - Format questions clearly with ### Q1:, ### Q2:, etc. - No extra text or explanations question_instructionsr4Trarz7Content and questions created and cleaned successfully.)r(dictrrrTrr8r*r"Levelmotiver}rUr'second_stage_crewrWrgrjrkrcontent_reviewerreview_and_enrich_content_clean_text_content qa_gen_crewrrirrr)r0r9r'rErFrGr:r,rHrIrJrrKrLrOduration_minutes_matchrYr;rnr= complete_datar^rortopic_sections updated_dataupdated_content_pathout_filerrrrVs    /        z)ContentCreationService._create_new_lessonc Csd|s|S||}gd}|D]}tj|d|tjtjBd}qtjdd|tjd}tjdd|tjd}ddl}t|j}|d g}|D]#}|dksS|d ksS|d krY| |qE||vrc| |qE| d qEd |}td d |}tdd|}| dd}| d}g} |D]} | } | r| dkr| | q| r| dkrqqd | }| S)zTClean text content by removing word counts, special characters, and trailing 'Total')rlrmrnrrrorrprNrqrrrsrrtrurvrxtotal)r~rrrrrzr{r|r}rrr8rrr) r0rrrrzallowed cleaned_charsrr cleaned_linesr stripped_linerrrr5sB           z*ContentCreationService._clean_text_contentc Cs|s|S|}gd}|D] }t|d|}q |d}g}hd}|D]1}|} | |vr0q#d} |D]} | | dsH| | dsH| | krLd} nq4| sT||q#d|}td d|}|S) zFRemove trailing metadata words and phrases from text - SIMPLER VERSION)z?(?:\n\s*)?[Tt][Oo][Tt][Aa][Ll]\s*(?:[Ww][Oo][Rr][Dd][Ss]?\s*)?$z:\n\s*[Tt][Oo][Tt][Aa][Ll]\s*\n\s*[Ww][Oo][Rr][Dd][Ss]?\s*$z\n\s*[Tt][Oo][Tt][Aa][Ll]\s*$z\n\s*[Ww][Oo][Rr][Dd][Ss]?\s*$z\n\s*\d+\s*(?:words?|total)\s*$rr>endwordcountrwordsr conclusionF:rTz2\n.*[Tt][Oo][Tt][Aa][Ll].*[Ww][Oo][Rr][Dd][Ss]?.*$)rrrrrrrr) r0r original_textpatternsrrrmetadata_wordsrstripped skip_linerrrrr~xs8      z0ContentCreationService._remove_trailing_metadataN)__name__ __module__ __qualname__r1r>rGr_rrrsrtrmryrrrrrrrlrSr/r!r\rUrjr`rgrVrr~rrrrr%Es<  $ICe4 79 GIi{T Cr%)"sysfastapirr utils.hashingruser_journey_service.crewr2user_journey_service.processors.duration_estimatorr5user_journey_service.processors.StagewiseCourseParserr0user_journey_service.processors.content_reviewerr8user_journey_service.processors.user_journey_synthesizerr *user_journey_service.tools.custom_stt_toolr duration_estimatorr synthesizerstt_toolpathlibr rrgttsr pydubr subprocessr$pydub_configuredr%rrrrs0           )