ڟicBdZddlZddlmZmZmZmZddlZddl m Z ddl m Z m Z mZmZmZddlmZej&eZGddZy) z9Interaction evaluator for preference learning evaluation.N)DictListAnyOptional) UserSimulator)accuracy precisionrecallf1compute_threshold)DiversitySamplerc HeZdZdZ d dedededefdZdZde e fd Z y ) InteractionEvaluatorz.Evaluator for interactive preference learning. simulatortop_kroundssampling_strategyc||_||_||_||_||_g|_g|_g|_g|_d|_ d|_ d|_ tjd|d|d|y)aYInitialize interaction evaluator. Args: manager: PipelineManager instance simulator: UserSimulator instance top_k: Number of patterns to recommend each round rounds: Number of interaction rounds sampling_strategy: Pattern sampling strategy - "top": Select top-k highest scores (original) - "mixed": Mix of top, middle, and random (recommended) - "stratified": Stratified sampling by score ranges - "uncertainty": Select patterns near decision boundary Nz(InteractionEvaluator initialized: top_k=z , rounds=z, sampling_strategy=)managerrrrrhistoryprecision_historyrecall_history f1_historypre_interaction_metrics all_patterns ground_truthloggerinfo)selfrrrrrs K/home/ubuntu/codebase/yexijia/保研/colocation_mvp/experiment/evaluator.py__init__zInteractionEvaluator.__init__s* "  !2 !# '+$!  >ugYvhVjk|j}~cddlm}m}|jjj ||}|Ft jdt|dtjj|dd|}n|jjj|}|s(|jjj|||S|||jjj j# }t jd t|dtjj|dd|jj$xr<|jj&duxr"|jj&j du} g} |D]} | j)} d | vr_| j+d g} t-| t.r| j1d } |jjj3| }||| d <| j5| | rbt j7dt|d|| ||jj&j |jj8}|St j7dt|d|| |d|jj8}|S)zScore patterns using manager's method which handles contrastive learning properly. This ensures that contrastive learning model is used when available. IMPORTANT: Use the UPDATED user vector that includes latest feedback. r)score_patternsbuild_user_vectorNzUsing fused user vector (dim=z, norm=.4f)user_idinteraction_round)dimz$Built user vector from profile (dim= embeddingpattern,zScoring z) patterns with contrastive learning model)modelalphaz5 patterns with baseline method (no contrastive model))preference.scorerr$r%rlearnerget_user_vectorrdebuglennplinalgnormmemoryget_user_profilescore_all_patternsembedderr/ get_sentence_embedding_dimensionstage3_use_contrastivetrainercopyget isinstancestrsplitencode_patternappendr stage3_alpha)rpatternsr)r*r$r% fused_vectoruser_vec user_profileuse_contrastivepatterns_with_embeddingsr- pattern_copy pattern_list pattern_vecscoress r _score_patterns_with_managerz1InteractionEvaluator._score_patterns_with_manager7s H||++;;GEVW  # LL8\9J8K7SUS\S\SaSabnSopsRttuv w#H <<..??HL||++>>#&7? )LL))//PPRH LL?H gVXV_V_VdVdemVnorUsstu v <<>>A,,..d:A,,..44D@  $& G"<<>L,.&{{9b9 lC0#/#5#5c#:L"ll33BB<P *0;L- $ + +L 9   KK(3x=/1Z[ \#(ll**00ll// F M KK(3x=/1fg h#(ll// F Mr"returnctjdtjd|jjj ddd|_tjdt |j dtjd |jj|j |_ tjd t|jjd t |jd |jj}|jjj|t|jj |jj"rt|jj"ngd } |jj%||j&}|d}tjd||j-|j |d}|.tj+ddgt |j z}t/|}t1|D cic]\}} || |k\rdnd} }} t3|j| t5|j| t7|j| t9|j| |d|_tjd|j:ddd|j:ddd|j:ddd |j:d!dd"|dd# t=|j&D]t} tjd$| dzd |j&d%| dk(rY|r%j?d&g} |j?d'i} n|jjA|d}|j?d&g} i} nf|r3d(tCvr'r%|j?d&g} |j?d'i} n1|jjA|d}|j?d&g} i} | s-tj+d)| dzd*|j dd+} i} g}| D]}|j?d,g}tE|tFr|jId-}d-jKtM|}| j?||j?dd} |jO| t | |jPk\r]tSjT| ||jP|jV.}tjd/t |d0|jVd1n1| }tjd2t |d3|jPd#ggd4}|D]}|jjY|}tE|j?d,tr!d-jK|j?d,gntG|j?d,d5}|dk(r|d6jO||d7jO|tjd8t |d6d9t |d7d:|rb| |j&dz krP |jj[||}tjd;t |j?d&gd<n4|r2 |jj[||tjd>|jj\<|jj\j^tjad@| dzdA|j-|j || dz}|5tj+dB| dzdCdgt |j z}|rtcjd|}tjd@| dzdD|jgddE|jiddF|jkddGtcjl|ddH|jod t/|}i}t1|D]\}} | |k\rdnd||<tdI|jD}tjd@| dzdJ|d t |dK|dd# t3|j|}t5|j|}t7|j|}t9|j|}|jpjO||jrjO||jtjO||jvjO|tjd@| dzdL|dd|dd|dd |dd"|dd# wtjdM|jpdNdd|jrdNdd|jtdNdd |jvdNd|jp|jr|jt|jv|j:dOS#t($r%}tj+d|d}Yd}~d}~wwxYwcc} }w#t($r'}tj+d=|d}d}Yd}~d}~wwxYw#t($r#}tj+d?|Yd}~d}~wwxYw)Pa/Run evaluation loop. Each round: 1. Recommend patterns 2. Simulate feedback 3. Update model 4. Predict ALL patterns 5. Compute accuracy 6. Store history Returns: List of accuracy scores for each round zStarting evaluation loop...zStep 1: Mining all patterns...g333333? confidence)min_participationmax_pattern_sizepriorityzMined z patternsz Step 2: Building ground truth...zGround truth: /z patterns likedz(I want to find good co-location patterns)iteration_rounds session_idzInitialized iteration session: z8Failed to start iteration, using process_query instead: Nrr(z8No pre-interaction scores available, using zero baselineg)rr r r thresholdz;Pre-interaction metrics (round 0, no feedback) - Accuracy: rr&z , Precision: r z , Recall: r z, F1: r z (threshold: r'z === Round z ===rHsimilarity_scores next_resultzNo patterns available in round z&, using top patterns from all_patternsr-r.)rstrategyz Selected z patterns using z sampling strategyz Using all z% available patterns (less than top_k=)positivenegativercrdzSimulated feedback: z positive, z negativez&Model updated via next_iteration, got z patterns for next roundz%Failed to update via next_iteration: z-Model updated via next_iteration (last round)z3Failed to update via next_iteration on last round: zRound z/: Trainer model available, using it for scoringzNo scores available in round z, using random baselinez score stats: min=z, max=z, mean=z , median=z, std=c3,K|] }|dk(s dyw)r]N).0vs r z+InteractionEvaluator.run..Ks&Q2FQ!q&q2Fs z predictions: z# predicted as positive (threshold: z metrics - Accuracy: z/Evaluation complete. Final metrics - Accuracy: )rr r r pre_interaction)?  67 NN==d>O>OP nS):):)A)A)C%D$EQs4K\K\G]F^^mno,,-- 44  %% &-1^^-D-DD(( )" ; !\\99%RVR]R]9^N' 5J KK9*F G 66   7   NNU VT%6%6!77J)*5 ( 3 3 U u -!1 43  !!2!2OD"4#4#4oFT..@T&&8& ( $   55jA#FG66{CCHI33H=cBC//5c:;(-Q  0 t{{#A KK*QqSE4;;-t< =Av*8*<*qseCZ[\T%6%6!77 hhv.  fQqSE);KOOrs>?,,.QQ/   8 $X X r"