[2026-10-17 15:28:05] Registered linear attn model: config=GigaChat35Config, backend=GDNAttnBackend, archs=['GigaChat35ForCausalLM', 'GigaChat35ForCausalLMNextN']
[2026-10-17 15:28:08] Attention backend not specified. Use flashinfer backend by default.
[2026-10-17 15:28:08] Max running requests is reset to 48 for speculative decoding. You can override this by explicitly setting --max-running-requests.
[2026-10-17 15:28:10] server_args={'model_path': 'Qwen/Qwen3-8B', 'tokenizer_path': 'Qwen/Qwen3-8B', 'tokenizer_mode': 'auto', 'tokenizer_backend': 'huggingface', 'tokenizer_worker_num': 1, 'detokenizer_worker_num': 1, 'skip_tokenizer_init': False, 'load_format': 'auto', 'model_loader_extra_config': '{}', 'trust_remote_code': False, 'context_length': None, 'is_embedding': False, 'enable_multimodal': None, 'revision': None, 'model_impl': 'auto', 'model_config_parser': 'auto', 'json_model_override_args': '{}', 'dtype': 'auto', 'quantization': None, 'quantization_param_path': None, 'kv_cache_dtype': 'auto', 'enable_fp32_lm_head': False, 'modelopt_quant': None, 'modelopt_checkpoint_restore_path': None, 'modelopt_checkpoint_save_path': None, 'modelopt_export_path': None, 'quantize_and_serve': False, 'rl_quant_profile': None, 'enable_tf32_matmul': False, 'mem_fraction_static': 0.85, 'max_running_requests': 48, 'max_queued_requests': None, 'max_total_tokens': None, 'chunked_prefill_size': 4096, 'prefill_decode_interval': 0, 'enable_dynamic_chunking': False, 'max_prefill_tokens': 16384, 'prefill_max_requests': None, 'schedule_policy': 'fcfs', 'enable_priority_scheduling': False, 'disable_priority_preemption': False, 'default_priority_value': None, 'abort_on_priority_when_disabled': False, 'schedule_low_priority_values_first': False, 'priority_scheduling_preemption_threshold': 10, 'retraction_policy': 'length', 'schedule_conservativeness': 1.0, 'page_size': 1, 'c128_page_size': 16, 'swa_full_tokens_ratio': 0.8, 'disable_hybrid_swa_memory': False, 'radix_eviction_policy': 'lru', 'radix_eviction_policy_config': None, 'prefill_only_disable_kv_cache': False, 'disable_radix_cache': False, 'enable_page_major_kv_layout': False, 'enable_unified_memory': False, 'disable_chunked_prefix_cache': False, 'disable_overlap_schedule': False, 'num_continuous_decode_steps': 1, 'scheduler_recv_interval': 1, 'enable_mixed_chunk': False, 'nccl_port': None, 'dist_timeout': None, 'dist_init_addr': None, 'gated_launch_port': None, 'nnodes': 1, 'node_rank': 0, 'tp_size': 1, 'dcp_size': 1, 'pp_size': 1, 'pp_max_micro_batch_size': None, 'pp_async_batch_depth': 0, 'dp_size': 1, 'attn_dp_size': 1, 'load_balance_method': 'round_robin', 'attn_cp_size': 1, 'moe_dp_size': 1, 'dwdp_size': 1, 'dcp_comm_backend': 'ag_rs', 'dcp_replicate_q_proj': None, 'enable_prefill_cp': False, 'cp_strategy': None, 'enable_dsa_cache_layer_split': False, 'enable_cp_decode_attn_tp': False, 'enable_kv_cache_sharding': False, 'enable_dp_attention': False, 'enable_dp_attention_local_control_broadcast': False, 'enable_dp_lm_head': False, 'enable_tp_lm_head_all_to_all': False, 'enable_attn_tp_input_scattered': False, 'enable_shared_experts_attn_tp': False, 'shared_experts_tp_size': None, 'enable_dense_mlp_attn_tp': False, 'enable_layernorm_sp': False, 'disable_attn_tp_gather': False, 'enable_p2p_check': False, 'device': 'cuda', 'base_gpu_id': 0, 'gpu_id_step': 1, 'random_seed': 245063579, 'mlx_enable_sampling': False, 'watchdog_timeout': 300, 'soft_watchdog_timeout': None, 'sleep_on_idle': False, 'use_ray': False, 'custom_sigquit_handler': None, 'numa_node': None, 'gc_threshold': None, 'host': '127.0.0.1', 'port': 30000, 'fastapi_root_path': '', 'smg_grpc_mode': False, 'grpc_mode': False, 'grpc_port': None, 'grpc_worker_threads': 4, 'sidecar': None, 'sidecar_args': None, 'skip_server_warmup': False, 'warmups': None, 'enable_http2': False, 'http2_max_concurrent_streams': 200, 'http2_initial_connection_window_size': 1048576, 'ssl_keyfile': None, 'ssl_certfile': None, 'ssl_ca_certs': None, 'ssl_keyfile_password': None, 'enable_ssl_refresh': False, 'api_key': None, 'admin_api_key': None, 'served_model_name': 'Qwen/Qwen3-8B', 'weight_version': 'default', 'chat_template': None, 'trust_request_chat_template': False, 'hf_chat_template_name': None, 'completion_template': None, 'file_storage_path': None, 'enable_cache_report': True, 'return_input_ids': False, 'return_output_ids': False, 'reasoning_parser': None, 'default_chat_template_kwargs': None, 'strip_thinking_cache': False, 'enable_strict_thinking': False, 'tool_call_parser': None, 'tool_server': None, 'sampling_defaults': 'model', 'asr_max_buffer_seconds': 60, 'asr_max_concurrent_sessions': 32, 'preferred_sampling_params': None, 'allow_auto_truncate': False, 'stream_interval': 1, 'batch_notify_size': 16, 'stream_response_default_include_usage': False, 'incremental_streaming_output': False, 'enable_streaming_session': False, 'enable_session_radix_cache': False, 'log_level': 'info', 'log_level_http': None, 'log_requests': False, 'log_requests_level': 2, 'log_requests_format': 'text', 'log_requests_target': None, 'uvicorn_access_log_exclude_prefixes': [], 'crash_dump_folder': None, 'show_time_cost': False, 'enable_metrics': True, 'smg_http_sidecar_port': None, 'enable_mfu_metrics': False, 'enable_metrics_for_all_schedulers': False, 'load_snapshot_publish_interval': 15, 'tokenizer_metrics_custom_labels_header': 'x-custom-labels', 'tokenizer_metrics_allowed_custom_labels': None, 'extra_metric_labels': None, 'bucket_time_to_first_token': None, 'bucket_inter_token_latency': None, 'bucket_e2e_request_latency': None, 'prompt_tokens_buckets': None, 'generation_tokens_buckets': None, 'gc_warning_threshold_secs': 0.0, 'decode_log_interval': 40, 'enable_request_time_stats_logging': False, 'kv_events_config': None, 'load_publish_endpoint': None, 'enable_forward_pass_metrics': False, 'forward_pass_metrics_worker_id': '', 'forward_pass_metrics_ipc_name': None, 'enable_trace': False, 'trace_modules': 'request', 'otlp_traces_endpoint': 'localhost:4317', 'export_metrics_to_file': False, 'export_metrics_to_file_dir': None, 'stat_loggers': None, 'constrained_json_whitespace_pattern': None, 'constrained_json_disable_any_whitespace': False, 'constrained_json_max_whitespace_cnt': None, 'attention_backend': 'flashinfer', 'decode_attention_backend': None, 'enable_lean_attention': None, 'prefill_attention_backend': None, 'sampling_backend': 'flashinfer', 'grammar_backend': 'xgrammar', 'radix_cache_backend': None, 'mm_attention_backend': None, 'fp8_gemm_runner_backend': 'auto', 'fp4_gemm_runner_backend': 'auto', 'bf16_gemm_backend': 'auto', 'dsa_prefill_backend': None, 'dsv4_prefill_backend': 'auto', 'dsa_decode_backend': None, 'dsa_paged_mqa_logits_backend': 'auto', 'dsa_topk_backend': 'sgl-kernel', 'disable_flashinfer_autotune': False, 'flashinfer_autotune_skip_ops': None, 'mamba_backend': 'triton', 'cuda_graph_config': {'decode': {'backend': 'full', 'max_bs': 48, 'bs': [1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 40, 44, 48], 'tc_compiler': 'eager', 'max_context_size': None, 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None, 'max_seq_len': None}, 'prefill': {'backend': 'breakable', 'max_bs': 4096, 'bs': [4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096], 'tc_compiler': 'eager', 'max_context_size': None, 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None, 'max_seq_len': None}}, 'cuda_graph_backend_decode': None, 'cuda_graph_backend_prefill': None, 'cuda_graph_max_bs_decode': None, 'cuda_graph_max_bs_prefill': None, 'cuda_graph_bs_decode': None, 'cuda_graph_bs_prefill': None, 'cuda_graph_prefill_max_context': None, 'disable_prefill_cuda_graph': False, 'disable_decode_cuda_graph': False, 'disable_cuda_graph': False, 'disable_cuda_graph_padding': False, 'enable_profile_cuda_graph': False, 'enable_cudagraph_gc': False, 'debug_cuda_graph': False, 'enable_layerwise_nvtx_marker': False, 'enable_nccl_nvls': False, 'enable_symm_mem': False, 'triton_attention_reduce_in_fp32': False, 'triton_attention_num_kv_splits': 8, 'triton_attention_split_tile_size': None, 'flashinfer_mla_disable_ragged': False, 'enable_fused_qk_norm_rope': False, 'enable_precise_embedding_interpolation': False, 'enable_fused_moe_sum_all_reduce': False, 'enable_deepseek_v4_fp4_indexer': False, 'disable_custom_all_reduce': False, 'enable_mscclpp': False, 'enable_torch_symm_mem': False, 'enable_scattered_sconv': False, 'pre_warm_nccl': False, 'boundary_reduction': 'ar', 'enable_quant_communications': False, 'enable_flashinfer_allreduce_fusion': False, 'enforce_disable_flashinfer_allreduce_fusion': False, 'flashinfer_allreduce_fusion_backend': None, 'enable_aiter_allreduce_fusion': False, 'enable_torch_compile': False, 'enable_torch_compile_debug_mode': False, 'torch_compile_max_bs': 32, 'speculative_algorithm': 'EAGLE3', 'uno_lora_path': None, 'speculative_draft_model_path': 'Tengyunw/qwen3_8b_eagle3', 'speculative_draft_model_revision': 'main', 'speculative_draft_load_format': None, 'speculative_num_steps': 3, 'speculative_eagle_topk': 1, 'speculative_num_draft_tokens': 4, 'speculative_dflash_block_size': None, 'speculative_dspark_block_size': None, 'speculative_dspark_sps_table_path': None, 'speculative_dspark_confidence_sts_path': None, 'speculative_dspark_align_verify_tokens_to_graph_tier': False, 'speculative_accept_threshold_single': 1.0, 'speculative_accept_threshold_acc': 1.0, 'speculative_use_rejection_sampling': False, 'speculative_token_map': None, 'speculative_attention_mode': 'prefill', 'speculative_draft_attention_backend': None, 'speculative_dsa_topk_backend': 'sgl-kernel', 'speculative_draft_kv_cache_dtype': None, 'speculative_draft_window_size': None, 'speculative_moe_runner_backend': 'auto', 'speculative_moe_a2a_backend': None, 'speculative_enable_w4a4_mxfp4_megamoe': None, 'speculative_draft_model_quantization': None, '_speculative_draft_quantization_explicitly_set': False, 'speculative_skip_dp_mlp_sync': False, 'enable_multi_layer_eagle': False, 'speculative_adaptive': False, 'speculative_adaptive_config': None, 'decoupled_spec_bind_endpoint': None, 'decoupled_spec_connect_endpoints': None, 'decoupled_spec_rank': None, 'decoupled_spec_role': 'null', 'spec_trace_dir': None, 'speculative_ngram_min_bfs_breadth': 1, 'speculative_ngram_max_bfs_breadth': 10, 'speculative_ngram_match_type': 'BFS', 'speculative_ngram_max_trie_depth': 18, 'speculative_ngram_capacity': 10000000, 'speculative_ngram_external_corpus_path': None, 'speculative_ngram_external_sam_budget': 0, 'speculative_ngram_external_corpus_max_tokens': 10000000, 'ep_size': 1, 'moe_a2a_backend': 'none', 'enable_w4a4_mxfp4_megamoe': False, 'deepep_v2_mode': 'direct', 'moe_runner_backend': 'auto', 'flashinfer_mxfp4_moe_precision': 'default', 'deepep_mode': 'auto', 'fuseep_mode': 2, 'deepep_dispatcher_output_dtype': 'auto', 'ep_num_redundant_experts': 0, 'ep_dispatch_algorithm': None, 'init_expert_location': 'trivial', 'enable_eplb': False, 'eplb_algorithm': 'auto', 'eplb_rebalance_num_iterations': 1000, 'eplb_rebalance_layers_per_chunk': None, 'eplb_min_rebalancing_utilization_threshold': 1.0, 'expert_distribution_recorder_mode': None, 'expert_distribution_recorder_buffer_size': 1000, 'expert_balancedness_report_mode': 'off', 'deepep_config': None, 'moe_dense_tp_size': None, 'elastic_ep_backend': None, 'enable_elastic_expert_backup': False, 'mooncake_ib_device': None, 'enable_waterfill': False, 'ep_join_mode': None, 'ep_join_rank_offset': 0, 'elastic_ep_initial_size': None, 'max_ep_size': None, 'elastic_ep_scale_timeout': 600, 'disable_flashinfer_cutlass_moe_fp4_allgather': False, 'disable_shared_experts_fusion': False, 'enforce_shared_experts_fusion': False, 'max_mamba_cache_size': None, 'mamba_ssm_dtype': None, 'mamba_max_states_per_path': -1, 'enable_mamba_cache_stochastic_rounding': False, 'mamba_cache_philox_rounds': 0, 'mamba_full_memory_ratio': 0.9, 'mamba_radix_cache_strategy': 'auto', 'uses_mamba_radix_cache': False, 'mamba_track_interval': 256, 'enable_int8_mamba_checkpoint': False, 'int8_mamba_ckpt_size': None, 'linear_attn_backend': 'triton', 'linear_attn_decode_backend': None, 'linear_attn_prefill_backend': None, 'linear_attn_verify_backend': None, 'enable_linear_replayssm': False, 'linear_replayssm_cache_len': 16, 'enable_linear_replayssm_spec': False, 'enable_hierarchical_cache': False, 'hicache_host_memory_mode': 'cache', 'hicache_ratio': 2.0, 'hicache_size': 0, 'hicache_write_policy': 'write_through', 'hicache_io_backend': 'kernel', 'hicache_mem_layout': 'page_first', 'hicache_storage_backend': None, 'hicache_storage_prefetch_policy': 'timeout', 'hicache_storage_backend_extra_config': None, 'hicache_storage_prefetch_retry_poll_interval': 8, 'hicache_storage_prefetch_retry_max_attempts': 8, 'enable_unified_cache_external_linker': False, 'unified_cache_external_linker_backend': 'mooncake', 'enable_hisparse': False, 'hisparse_config': None, 'enable_broadcast_mm_inputs_process': False, 'enable_prefix_mm_cache': False, 'mm_enable_dp_encoder': False, 'mm_process_config': {}, 'mm_processor_worker_num': 0, 'mm_io_worker_num': 0, 'allowed_media_domains': [], 'media_url_max_file_size_mb': 64, 'mm_preprocess_cache_size_mb': None, 'trust_mm_content_hashes': False, 'limit_mm_data_per_request': None, 'enable_mm_global_cache': False, 'image_processor_backend': 'auto', 'mm_global_cache_backend': 'mooncake', 'disable_fast_image_processor': False, 'mm_feature_transport': 'cpu', 'enable_lora': None, 'enable_lora_overlap_loading': None, 'max_lora_rank': None, 'lora_target_modules': None, 'lora_paths': None, 'max_loaded_loras': None, 'max_loras_per_batch': 8, 'lora_eviction_policy': 'lru', 'lora_backend': 'csgmv', 'max_lora_chunk_size': 16, 'experts_shared_outer_loras': None, 'lora_use_virtual_experts': False, 'lora_strict_loading': False, 'lora_drain_wait_threshold': 0.0, 'enable_two_batch_overlap': False, 'enable_single_batch_overlap': False, 'tbo_token_distribution_threshold': 0.48, 'cpu_offload_gb': 0, 'offload_group_size': -1, 'offload_num_in_group': 1, 'offload_prefetch_step': 1, 'offload_mode': 'cpu', 'enable_lmcache': False, 'lmcache_config_file': None, 'enable_flexkv': False, 'flexkv_config_file': None, 'kt_weight_path': None, 'kt_method': 'AMXINT4', 'kt_cpuinfer': None, 'kt_threadpool_count': 2, 'kt_num_gpu_experts': None, 'kt_max_deferred_experts_per_token': None, 'dllm_algorithm': None, 'dllm_algorithm_config': None, 'dllm_fdfo': True, 'disaggregation_mode': 'null', 'disaggregation_transfer_backend': 'mooncake', 'disaggregation_bootstrap_port': 8998, 'disaggregation_ib_device': None, 'disaggregation_decode_enable_radix_cache': False, 'disaggregation_decode_enable_offload_kvcache': False, 'disaggregation_decode_retraction_backup': None, 'num_reserved_decode_tokens': 512, 'disaggregation_decode_extra_slots': None, 'disaggregation_decode_polling_interval': 1, 'optimistic_prefill_attempts': 0, 'encoder_only': False, 'language_only': False, 'language_model_only': False, 'encoder_transfer_backend': 'zmq_to_scheduler', 'encoder_urls': [], 'encoder_bootstrap_port': 8997, 'encoder_register_urls': [], 'enable_adaptive_dispatch_to_encoder': False, 'enable_pdmux': False, 'pdmux_config_path': None, 'sm_group_num': 8, 'startup_weight_load_mode': 'serial', 'custom_weight_loader': [], 'weight_loader_disable_mmap': False, 'weight_loader_prefetch_checkpoints': False, 'weight_loader_prefetch_num_threads': 4, 'weight_loader_drop_cache_after_load': False, 'remote_instance_weight_loader_seed_instance_ip': None, 'remote_instance_weight_loader_seed_instance_service_port': None, 'remote_instance_weight_loader_send_weights_group_ports': None, 'remote_instance_weight_loader_backend': 'nccl', 'remote_instance_weight_loader_start_seed_via_transfer_engine': False, 'engine_info_bootstrap_port': 6789, 'modelexpress_config': None, 'download_dir': None, 'model_checksum': None, 'delete_ckpt_after_loading': False, 'decrypted_config_file': None, 'decrypted_draft_config_file': None, 'checkpoint_engine_wait_weights_before_ready': False, 'enable_prefill_delayer': False, 'prefill_delayer_max_delay_passes': 30, 'prefill_delayer_token_usage_low_watermark': None, 'prefill_delayer_forward_passes_buckets': None, 'prefill_delayer_wait_seconds_buckets': None, 'prefill_delayer_queue_min_ratio': None, 'prefill_delayer_max_delay_ms': None, 'min_free_slots_delay': None, 'enable_deterministic_inference': False, 'rl_on_policy_target': None, 'kv_canary': 'none', 'kv_canary_real_data': 'none', 'kv_canary_sweep_interval': 0, 'enable_dynamic_batch_tokenizer': False, 'dynamic_batch_tokenizer_batch_size': 32, 'dynamic_batch_tokenizer_batch_timeout': 0.002, 'enable_tokenizer_batch_encode': False, 'disable_tokenizer_batch_decode': False, 'debug_tensor_dump_output_folder': None, 'debug_tensor_dump_layers': None, 'debug_tensor_dump_input_file': None, 'enable_memory_saver': False, 'enable_weights_cpu_backup': False, 'enable_draft_weights_cpu_backup': False, 'enable_custom_logit_processor': False, 'enable_return_hidden_states': False, 'return_hidden_states_mode': None, 'enable_return_routed_experts': False, 'enable_return_indexer_topk': False, 'sampling_mask_max_tokens': 4096, 'disable_outlines_disk_cache': False, 'enable_mis': False, 'weight_cache_mode': 'off', 'weight_cache_socket': None, 'weight_cache_timeout': 1800, 'forward_hooks': None, 'msprobe_dump_config': None, 'ple_offload_embedding': None, 'ple_offload_backend': 'pinned', 'ple_offload_dir': None, 'disaggregation_enable_kv_checksum': False, 'enable_pd_role_switch': False, 'speculative_domino_candidate_pool_size': 2048, 'flashinfer_a2a_dispatch_type': None, 'cuda_graph_max_seq_len_prefill': None, 'prefill_kv_cache_dequant_dtype': 'auto', 'dsv4_attn_backend': 'auto', 'otlp_service_name': None, '_radix_eviction_policy_explicitly_set': False, 'hicache_host_memory_fraction': 0.8, 'enable_linker_mla_dedup': False, '_swa_full_tokens_ratio_explicitly_set': False, 'swa_prefix_tails': None, 'enable_encoder_swa_bounded_replay': False, 'enable_decoder_swa_bounded_replay': False, 'enable_response_store': False, 'disaggregation_decode_host_receive_threshold': 1.0, 'qsa_indexer_dtype': 'auto', 'disaggregation_decode_allocation_policy': 'early', 'speculative_boundary_reduction': 'rs+rsv', 'speculative_use_block_verification': False, 'speculative_draft_sink_size': None, 'disable_aiter_allreduce_fusion_in_prefill': False, 'disable_aiter_allreduce_fusion_in_decode': False, 'dsa_triton_union': 0, 'enable_dsa_fused_indexer': None, 'enable_cp_tp_group_sharing': False, 'enable_watermark': False, 'watermark_config': '<redacted>', 'watermark_key': '<redacted>', 'watermark_key_b': '<redacted>', 'watermark_context_window': 4, 'watermark_mixing_probability': 0.5, 'watermark_max_probability': 1.0, 'watermark_default_enabled': False, 'watermark_enforce_all': False, 'grpc_response_timeout_secs': 300}
[2026-10-17 15:28:11] Using default HuggingFace chat template with detected content format: string
[2026-10-17 15:28:12] Auto-detected template features: reasoning_config=ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None), reasoning_parser=qwen3, tool_call_parser=qwen
[2026-10-17 15:28:20] Init parallel begin.
[2026-10-17 15:28:20] Init parallel ends. elapsed=0.21 s
[2026-10-17 15:28:21] Memory baseline taken. mem usage=0.00 GB
/data/sglang/.venv/lib/python3.12/site-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`.
  warnings.warn(
/data/sglang/.venv/lib/python3.12/site-packages/torch/jit/_script.py:1643: FutureWarning: `torch.jit.interface` is deprecated. Please use `torch.compile` instead.
  warnings.warn(
/data/sglang/.venv/lib/python3.12/site-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`.
  warnings.warn(
[2026-10-17 15:28:23] Load weight begin. avail mem=21.69 GB
[2026-10-17 15:28:23] Found local HF snapshot for Qwen/Qwen3-8B at /data/sglang/hf/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218; skipping download.

Multi-thread loading shards:   0% Completed | 0/5 [00:00<?, ?it/s]
Multi-thread loading shards:  20% Completed | 1/5 [00:00<00:02,  1.35it/s]
Multi-thread loading shards:  40% Completed | 2/5 [00:01<00:02,  1.19it/s]
Multi-thread loading shards:  60% Completed | 3/5 [00:02<00:01,  1.21it/s]
Multi-thread loading shards:  80% Completed | 4/5 [00:03<00:00,  1.29it/s]
Multi-thread loading shards: 100% Completed | 5/5 [00:03<00:00,  1.63it/s]
Multi-thread loading shards: 100% Completed | 5/5 [00:03<00:00,  1.43it/s]
[2026-10-17 15:28:26] Load weight end. elapsed=3.68 s, type=Qwen3ForCausalLM, avail mem=6.41 GB, mem usage=15.28 GB.
[2026-10-17 15:28:27] SPECULATIVE_MOE_RUNNER_BACKEND is not initialized, using auto backend
[2026-10-17 15:28:27] SPECULATIVE_MOE_A2A_BACKEND is not initialized, using none backend
[2026-10-17 15:28:27] Memory baseline taken. mem usage=0.00 GB
[2026-10-17 15:28:27] Load weight begin. avail mem=6.41 GB
[2026-10-17 15:28:27] Found local HF snapshot for Tengyunw/qwen3_8b_eagle3 at /data/sglang/hf/hub/models--Tengyunw--qwen3_8b_eagle3/snapshots/2a1059d51f622b8cad7d7d72840153ffea5488a0; skipping download.

Multi-thread loading pt checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Multi-thread loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.74it/s]
Multi-thread loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.74it/s]
[2026-10-17 15:28:28] Load weight end. elapsed=0.67 s, type=LlamaForCausalLMEagle3, avail mem=4.50 GB, mem usage=1.90 GB.
[2026-10-17 15:28:28] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 8827, K size: 0.61 GB, V size: 0.61 GB
[2026-10-17 15:28:28] Memory pool end. avail mem=3.22 GB
[2026-10-17 15:28:28] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 8827, K size: 0.02 GB, V size: 0.02 GB
[2026-10-17 15:28:28] Memory pool end. avail mem=3.18 GB
[2026-10-17 15:28:29] Capture target prefill CUDA graph begin. backend=breakable, num_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096], avail mem=3.71 GB

  0%|          | 0/50 [00:00<?, ?it/s]
Capturing prefill shape (num_tokens=4096 avail_mem=3.71 GB):   0%|          | 0/50 [00:00<?, ?it/s]
Capturing prefill shape (num_tokens=4096 avail_mem=3.71 GB):   2%|▏         | 1/50 [00:02<01:48,  2.21s/it]
Capturing prefill shape (num_tokens=3840 avail_mem=2.22 GB):   2%|▏         | 1/50 [00:02<01:48,  2.21s/it]
Capturing prefill shape (num_tokens=3840 avail_mem=2.22 GB):   4%|▍         | 2/50 [00:05<02:26,  3.06s/it]
Capturing prefill shape (num_tokens=3584 avail_mem=2.13 GB):   4%|▍         | 2/50 [00:05<02:26,  3.06s/it]
Capturing prefill shape (num_tokens=3584 avail_mem=2.13 GB):   6%|▌         | 3/50 [00:07<01:45,  2.25s/it]
Capturing prefill shape (num_tokens=3328 avail_mem=2.13 GB):   6%|▌         | 3/50 [00:07<01:45,  2.25s/it]
Capturing prefill shape (num_tokens=3328 avail_mem=2.13 GB):   8%|▊         | 4/50 [00:11<02:13,  2.90s/it]
Capturing prefill shape (num_tokens=3072 avail_mem=2.12 GB):   8%|▊         | 4/50 [00:11<02:13,  2.90s/it]
Capturing prefill shape (num_tokens=3072 avail_mem=2.12 GB):  10%|█         | 5/50 [00:12<01:43,  2.31s/it]
Capturing prefill shape (num_tokens=2816 avail_mem=2.11 GB):  10%|█         | 5/50 [00:12<01:43,  2.31s/it]
Capturing prefill shape (num_tokens=2816 avail_mem=2.11 GB):  12%|█▏        | 6/50 [00:15<02:00,  2.74s/it]
Capturing prefill shape (num_tokens=2560 avail_mem=2.11 GB):  12%|█▏        | 6/50 [00:15<02:00,  2.74s/it]
Capturing prefill shape (num_tokens=2560 avail_mem=2.11 GB):  14%|█▍        | 7/50 [00:17<01:49,  2.54s/it]
Capturing prefill shape (num_tokens=2304 avail_mem=2.10 GB):  14%|█▍        | 7/50 [00:17<01:49,  2.54s/it]
Capturing prefill shape (num_tokens=2304 avail_mem=2.10 GB):  16%|█▌        | 8/50 [00:19<01:26,  2.05s/it]
Capturing prefill shape (num_tokens=2048 avail_mem=2.10 GB):  16%|█▌        | 8/50 [00:19<01:26,  2.05s/it]
Capturing prefill shape (num_tokens=2048 avail_mem=2.10 GB):  18%|█▊        | 9/50 [00:21<01:34,  2.31s/it]
Capturing prefill shape (num_tokens=1792 avail_mem=2.09 GB):  18%|█▊        | 9/50 [00:21<01:34,  2.31s/it]
Capturing prefill shape (num_tokens=1792 avail_mem=2.09 GB):  20%|██        | 10/50 [00:22<01:15,  1.89s/it]
Capturing prefill shape (num_tokens=1536 avail_mem=2.09 GB):  20%|██        | 10/50 [00:22<01:15,  1.89s/it]
Capturing prefill shape (num_tokens=1536 avail_mem=2.09 GB):  22%|██▏       | 11/50 [00:25<01:19,  2.03s/it]
Capturing prefill shape (num_tokens=1280 avail_mem=2.08 GB):  22%|██▏       | 11/50 [00:25<01:19,  2.03s/it]
Capturing prefill shape (num_tokens=1280 avail_mem=2.08 GB):  24%|██▍       | 12/50 [00:26<01:07,  1.79s/it]
Capturing prefill shape (num_tokens=1024 avail_mem=2.07 GB):  24%|██▍       | 12/50 [00:26<01:07,  1.79s/it]
Capturing prefill shape (num_tokens=1024 avail_mem=2.07 GB):  26%|██▌       | 13/50 [00:26<00:52,  1.41s/it]
Capturing prefill shape (num_tokens=960 avail_mem=2.07 GB):  26%|██▌       | 13/50 [00:26<00:52,  1.41s/it] 
Capturing prefill shape (num_tokens=960 avail_mem=2.07 GB):  28%|██▊       | 14/50 [00:28<00:50,  1.41s/it]
Capturing prefill shape (num_tokens=896 avail_mem=2.06 GB):  28%|██▊       | 14/50 [00:28<00:50,  1.41s/it]
Capturing prefill shape (num_tokens=896 avail_mem=2.06 GB):  30%|███       | 15/50 [00:28<00:39,  1.13s/it]
Capturing prefill shape (num_tokens=832 avail_mem=2.06 GB):  30%|███       | 15/50 [00:28<00:39,  1.13s/it]
Capturing prefill shape (num_tokens=832 avail_mem=2.06 GB):  32%|███▏      | 16/50 [00:30<00:40,  1.18s/it]
Capturing prefill shape (num_tokens=768 avail_mem=2.05 GB):  32%|███▏      | 16/50 [00:30<00:40,  1.18s/it]
Capturing prefill shape (num_tokens=768 avail_mem=2.05 GB):  34%|███▍      | 17/50 [00:30<00:31,  1.05it/s]
Capturing prefill shape (num_tokens=704 avail_mem=2.04 GB):  34%|███▍      | 17/50 [00:30<00:31,  1.05it/s]
Capturing prefill shape (num_tokens=704 avail_mem=2.04 GB):  36%|███▌      | 18/50 [00:31<00:31,  1.02it/s]
Capturing prefill shape (num_tokens=640 avail_mem=2.04 GB):  36%|███▌      | 18/50 [00:31<00:31,  1.02it/s]
Capturing prefill shape (num_tokens=640 avail_mem=2.04 GB):  38%|███▊      | 19/50 [00:32<00:26,  1.15it/s]
Capturing prefill shape (num_tokens=576 avail_mem=2.03 GB):  38%|███▊      | 19/50 [00:32<00:26,  1.15it/s]
Capturing prefill shape (num_tokens=576 avail_mem=2.03 GB):  40%|████      | 20/50 [00:32<00:21,  1.41it/s]
Capturing prefill shape (num_tokens=512 avail_mem=2.03 GB):  40%|████      | 20/50 [00:32<00:21,  1.41it/s]
Capturing prefill shape (num_tokens=512 avail_mem=2.03 GB):  42%|████▏     | 21/50 [00:33<00:21,  1.38it/s]
Capturing prefill shape (num_tokens=480 avail_mem=2.02 GB):  42%|████▏     | 21/50 [00:33<00:21,  1.38it/s]
Capturing prefill shape (num_tokens=480 avail_mem=2.02 GB):  44%|████▍     | 22/50 [00:33<00:16,  1.72it/s]
Capturing prefill shape (num_tokens=448 avail_mem=2.02 GB):  44%|████▍     | 22/50 [00:33<00:16,  1.72it/s]
Capturing prefill shape (num_tokens=448 avail_mem=2.02 GB):  46%|████▌     | 23/50 [00:34<00:17,  1.57it/s]
Capturing prefill shape (num_tokens=416 avail_mem=2.01 GB):  46%|████▌     | 23/50 [00:34<00:17,  1.57it/s]
Capturing prefill shape (num_tokens=416 avail_mem=2.01 GB):  48%|████▊     | 24/50 [00:34<00:13,  1.94it/s]
Capturing prefill shape (num_tokens=384 avail_mem=2.00 GB):  48%|████▊     | 24/50 [00:34<00:13,  1.94it/s]
Capturing prefill shape (num_tokens=384 avail_mem=2.00 GB):  50%|█████     | 25/50 [00:35<00:14,  1.78it/s]
Capturing prefill shape (num_tokens=352 avail_mem=2.00 GB):  50%|█████     | 25/50 [00:35<00:14,  1.78it/s]
Capturing prefill shape (num_tokens=352 avail_mem=2.00 GB):  52%|█████▏    | 26/50 [00:35<00:12,  1.93it/s]
Capturing prefill shape (num_tokens=320 avail_mem=1.99 GB):  52%|█████▏    | 26/50 [00:35<00:12,  1.93it/s]
Capturing prefill shape (num_tokens=320 avail_mem=1.99 GB):  54%|█████▍    | 27/50 [00:35<00:10,  2.27it/s]
Capturing prefill shape (num_tokens=288 avail_mem=1.99 GB):  54%|█████▍    | 27/50 [00:35<00:10,  2.27it/s]
Capturing prefill shape (num_tokens=288 avail_mem=1.99 GB):  56%|█████▌    | 28/50 [00:36<00:10,  2.06it/s]
Capturing prefill shape (num_tokens=256 avail_mem=1.98 GB):  56%|█████▌    | 28/50 [00:36<00:10,  2.06it/s]
Capturing prefill shape (num_tokens=256 avail_mem=1.98 GB):  58%|█████▊    | 29/50 [00:36<00:08,  2.53it/s]
Capturing prefill shape (num_tokens=240 avail_mem=1.98 GB):  58%|█████▊    | 29/50 [00:36<00:08,  2.53it/s]
Capturing prefill shape (num_tokens=240 avail_mem=1.98 GB):  60%|██████    | 30/50 [00:37<00:07,  2.64it/s]
Capturing prefill shape (num_tokens=224 avail_mem=1.97 GB):  60%|██████    | 30/50 [00:37<00:07,  2.64it/s]
Capturing prefill shape (num_tokens=224 avail_mem=1.97 GB):  62%|██████▏   | 31/50 [00:37<00:06,  3.06it/s]
Capturing prefill shape (num_tokens=208 avail_mem=1.96 GB):  62%|██████▏   | 31/50 [00:37<00:06,  3.06it/s]
Capturing prefill shape (num_tokens=208 avail_mem=1.96 GB):  64%|██████▍   | 32/50 [00:37<00:05,  3.50it/s]
Capturing prefill shape (num_tokens=192 avail_mem=1.96 GB):  64%|██████▍   | 32/50 [00:37<00:05,  3.50it/s]
Capturing prefill shape (num_tokens=192 avail_mem=1.96 GB):  66%|██████▌   | 33/50 [00:37<00:05,  3.30it/s]
Capturing prefill shape (num_tokens=176 avail_mem=1.95 GB):  66%|██████▌   | 33/50 [00:37<00:05,  3.30it/s]
Capturing prefill shape (num_tokens=176 avail_mem=1.95 GB):  68%|██████▊   | 34/50 [00:37<00:04,  3.78it/s]
Capturing prefill shape (num_tokens=160 avail_mem=1.95 GB):  68%|██████▊   | 34/50 [00:37<00:04,  3.78it/s]
Capturing prefill shape (num_tokens=160 avail_mem=1.95 GB):  70%|███████   | 35/50 [00:38<00:04,  3.51it/s]
Capturing prefill shape (num_tokens=144 avail_mem=1.94 GB):  70%|███████   | 35/50 [00:38<00:04,  3.51it/s]
Capturing prefill shape (num_tokens=144 avail_mem=1.94 GB):  72%|███████▏  | 36/50 [00:38<00:03,  4.02it/s]
Capturing prefill shape (num_tokens=128 avail_mem=1.94 GB):  72%|███████▏  | 36/50 [00:38<00:03,  4.02it/s]
Capturing prefill shape (num_tokens=128 avail_mem=1.94 GB):  74%|███████▍  | 37/50 [00:38<00:03,  4.00it/s]
Capturing prefill shape (num_tokens=112 avail_mem=1.91 GB):  74%|███████▍  | 37/50 [00:38<00:03,  4.00it/s]
Capturing prefill shape (num_tokens=112 avail_mem=1.91 GB):  76%|███████▌  | 38/50 [00:38<00:02,  4.56it/s]
Capturing prefill shape (num_tokens=96 avail_mem=1.90 GB):  76%|███████▌  | 38/50 [00:38<00:02,  4.56it/s] 
Capturing prefill shape (num_tokens=96 avail_mem=1.90 GB):  78%|███████▊  | 39/50 [00:38<00:02,  5.20it/s]
Capturing prefill shape (num_tokens=80 avail_mem=1.90 GB):  78%|███████▊  | 39/50 [00:38<00:02,  5.20it/s]
Capturing prefill shape (num_tokens=80 avail_mem=1.90 GB):  80%|████████  | 40/50 [00:39<00:01,  5.42it/s]
Capturing prefill shape (num_tokens=64 avail_mem=1.89 GB):  80%|████████  | 40/50 [00:39<00:01,  5.42it/s]
Capturing prefill shape (num_tokens=64 avail_mem=1.89 GB):  82%|████████▏ | 41/50 [00:39<00:01,  5.63it/s]
Capturing prefill shape (num_tokens=48 avail_mem=1.89 GB):  82%|████████▏ | 41/50 [00:39<00:01,  5.63it/s]
Capturing prefill shape (num_tokens=48 avail_mem=1.89 GB):  84%|████████▍ | 42/50 [00:39<00:01,  5.84it/s]
Capturing prefill shape (num_tokens=32 avail_mem=1.88 GB):  84%|████████▍ | 42/50 [00:39<00:01,  5.84it/s]
Capturing prefill shape (num_tokens=32 avail_mem=1.88 GB):  86%|████████▌ | 43/50 [00:39<00:01,  6.36it/s]
Capturing prefill shape (num_tokens=28 avail_mem=1.87 GB):  86%|████████▌ | 43/50 [00:39<00:01,  6.36it/s]
Capturing prefill shape (num_tokens=28 avail_mem=1.87 GB):  88%|████████▊ | 44/50 [00:39<00:00,  6.17it/s]
Capturing prefill shape (num_tokens=24 avail_mem=1.87 GB):  88%|████████▊ | 44/50 [00:39<00:00,  6.17it/s]
Capturing prefill shape (num_tokens=24 avail_mem=1.87 GB):  90%|█████████ | 45/50 [00:39<00:00,  6.44it/s]
Capturing prefill shape (num_tokens=20 avail_mem=1.86 GB):  90%|█████████ | 45/50 [00:39<00:00,  6.44it/s]
Capturing prefill shape (num_tokens=20 avail_mem=1.86 GB):  92%|█████████▏| 46/50 [00:40<00:00,  6.84it/s]
Capturing prefill shape (num_tokens=16 avail_mem=1.86 GB):  92%|█████████▏| 46/50 [00:40<00:00,  6.84it/s]
Capturing prefill shape (num_tokens=16 avail_mem=1.86 GB):  94%|█████████▍| 47/50 [00:40<00:00,  6.07it/s]
Capturing prefill shape (num_tokens=12 avail_mem=1.85 GB):  94%|█████████▍| 47/50 [00:40<00:00,  6.07it/s]
Capturing prefill shape (num_tokens=12 avail_mem=1.85 GB):  96%|█████████▌| 48/50 [00:40<00:00,  6.63it/s]
Capturing prefill shape (num_tokens=8 avail_mem=1.85 GB):  96%|█████████▌| 48/50 [00:40<00:00,  6.63it/s] 
Capturing prefill shape (num_tokens=8 avail_mem=1.85 GB):  98%|█████████▊| 49/50 [00:40<00:00,  6.67it/s]
Capturing prefill shape (num_tokens=4 avail_mem=1.84 GB):  98%|█████████▊| 49/50 [00:40<00:00,  6.67it/s]
Capturing prefill shape (num_tokens=4 avail_mem=1.84 GB): 100%|██████████| 50/50 [00:40<00:00,  7.09it/s]
Capturing prefill shape (num_tokens=4 avail_mem=1.84 GB): 100%|██████████| 50/50 [00:40<00:00,  1.23it/s]
[2026-10-17 15:29:11] Capture target prefill CUDA graph end. elapsed=41.43 s, mem usage=1.17 GB, avail mem=2.54 GB.
[2026-10-17 15:29:11] Capture target verify CUDA graph begin. backend=full, num_tokens_per_req=4, avail mem=2.54 GB

  0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=48 avail_mem=2.49 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=48 avail_mem=2.49 GB):   4%|▍         | 1/23 [00:00<00:05,  3.83it/s]
Capturing batches (bs=44 avail_mem=2.49 GB):   4%|▍         | 1/23 [00:00<00:05,  3.83it/s]
Capturing batches (bs=44 avail_mem=2.49 GB):   9%|▊         | 2/23 [00:00<00:04,  4.22it/s]
Capturing batches (bs=40 avail_mem=2.46 GB):   9%|▊         | 2/23 [00:00<00:04,  4.22it/s]
Capturing batches (bs=40 avail_mem=2.46 GB):  13%|█▎        | 3/23 [00:00<00:04,  4.39it/s]
Capturing batches (bs=32 avail_mem=2.46 GB):  13%|█▎        | 3/23 [00:00<00:04,  4.39it/s]
Capturing batches (bs=32 avail_mem=2.46 GB):  17%|█▋        | 4/23 [00:00<00:04,  4.70it/s]
Capturing batches (bs=30 avail_mem=2.44 GB):  17%|█▋        | 4/23 [00:00<00:04,  4.70it/s]
Capturing batches (bs=30 avail_mem=2.44 GB):  22%|██▏       | 5/23 [00:01<00:03,  4.90it/s]
Capturing batches (bs=28 avail_mem=2.43 GB):  22%|██▏       | 5/23 [00:01<00:03,  4.90it/s]
Capturing batches (bs=28 avail_mem=2.43 GB):  26%|██▌       | 6/23 [00:01<00:03,  5.04it/s]
Capturing batches (bs=26 avail_mem=2.41 GB):  26%|██▌       | 6/23 [00:01<00:03,  5.04it/s]
Capturing batches (bs=26 avail_mem=2.41 GB):  30%|███       | 7/23 [00:01<00:04,  3.80it/s]
Capturing batches (bs=24 avail_mem=2.41 GB):  30%|███       | 7/23 [00:01<00:04,  3.80it/s]
Capturing batches (bs=24 avail_mem=2.41 GB):  35%|███▍      | 8/23 [00:01<00:03,  3.78it/s]
Capturing batches (bs=22 avail_mem=2.38 GB):  35%|███▍      | 8/23 [00:01<00:03,  3.78it/s]
Capturing batches (bs=22 avail_mem=2.38 GB):  39%|███▉      | 9/23 [00:02<00:03,  4.14it/s]
Capturing batches (bs=20 avail_mem=2.38 GB):  39%|███▉      | 9/23 [00:02<00:03,  4.14it/s]
Capturing batches (bs=20 avail_mem=2.38 GB):  43%|████▎     | 10/23 [00:02<00:02,  4.46it/s]
Capturing batches (bs=18 avail_mem=2.35 GB):  43%|████▎     | 10/23 [00:02<00:02,  4.46it/s]
Capturing batches (bs=18 avail_mem=2.35 GB):  48%|████▊     | 11/23 [00:02<00:02,  4.73it/s]
Capturing batches (bs=16 avail_mem=2.35 GB):  48%|████▊     | 11/23 [00:02<00:02,  4.73it/s]
Capturing batches (bs=16 avail_mem=2.35 GB):  52%|█████▏    | 12/23 [00:02<00:02,  4.95it/s]
Capturing batches (bs=14 avail_mem=2.32 GB):  52%|█████▏    | 12/23 [00:02<00:02,  4.95it/s]
Capturing batches (bs=14 avail_mem=2.32 GB):  57%|█████▋    | 13/23 [00:02<00:01,  5.13it/s]
Capturing batches (bs=12 avail_mem=2.32 GB):  57%|█████▋    | 13/23 [00:02<00:01,  5.13it/s]
Capturing batches (bs=12 avail_mem=2.32 GB):  61%|██████    | 14/23 [00:03<00:01,  5.26it/s]
Capturing batches (bs=10 avail_mem=2.30 GB):  61%|██████    | 14/23 [00:03<00:01,  5.26it/s]
Capturing batches (bs=10 avail_mem=2.30 GB):  65%|██████▌   | 15/23 [00:03<00:01,  5.36it/s]
Capturing batches (bs=8 avail_mem=2.29 GB):  65%|██████▌   | 15/23 [00:03<00:01,  5.36it/s] 
Capturing batches (bs=8 avail_mem=2.29 GB):  70%|██████▉   | 16/23 [00:03<00:01,  5.44it/s]
Capturing batches (bs=7 avail_mem=2.27 GB):  70%|██████▉   | 16/23 [00:03<00:01,  5.44it/s]
Capturing batches (bs=7 avail_mem=2.27 GB):  74%|███████▍  | 17/23 [00:03<00:01,  5.49it/s]
Capturing batches (bs=6 avail_mem=2.27 GB):  74%|███████▍  | 17/23 [00:03<00:01,  5.49it/s]
Capturing batches (bs=6 avail_mem=2.27 GB):  78%|███████▊  | 18/23 [00:03<00:00,  5.51it/s]
Capturing batches (bs=5 avail_mem=2.24 GB):  78%|███████▊  | 18/23 [00:03<00:00,  5.51it/s]
Capturing batches (bs=5 avail_mem=2.24 GB):  83%|████████▎ | 19/23 [00:03<00:00,  5.56it/s]
Capturing batches (bs=4 avail_mem=2.24 GB):  83%|████████▎ | 19/23 [00:03<00:00,  5.56it/s]
Capturing batches (bs=4 avail_mem=2.24 GB):  87%|████████▋ | 20/23 [00:04<00:00,  5.70it/s]
Capturing batches (bs=3 avail_mem=2.21 GB):  87%|████████▋ | 20/23 [00:04<00:00,  5.70it/s]
Capturing batches (bs=3 avail_mem=2.21 GB):  91%|█████████▏| 21/23 [00:04<00:00,  5.80it/s]
Capturing batches (bs=2 avail_mem=2.21 GB):  91%|█████████▏| 21/23 [00:04<00:00,  5.80it/s]
Capturing batches (bs=2 avail_mem=2.21 GB):  96%|█████████▌| 22/23 [00:04<00:00,  5.90it/s]
Capturing batches (bs=1 avail_mem=2.19 GB):  96%|█████████▌| 22/23 [00:04<00:00,  5.90it/s]
Capturing batches (bs=1 avail_mem=2.19 GB): 100%|██████████| 23/23 [00:04<00:00,  5.98it/s]
Capturing batches (bs=1 avail_mem=2.19 GB): 100%|██████████| 23/23 [00:04<00:00,  5.03it/s]
[2026-10-17 15:29:16] Capture target verify CUDA graph end. bs=[1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 40, 44, 48], elapsed=5.37 s, mem usage=0.36 GB, avail mem=2.18 GB.
[2026-10-17 15:29:16] Capture draft prefill CUDA graph begin. backend=breakable, num_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096], avail mem=2.18 GB

  0%|          | 0/50 [00:00<?, ?it/s]
Capturing prefill shape (num_tokens=4096 avail_mem=2.06 GB):   0%|          | 0/50 [00:00<?, ?it/s]
Capturing prefill shape (num_tokens=3840 avail_mem=1.51 GB):   0%|          | 0/50 [00:00<?, ?it/s]
Capturing prefill shape (num_tokens=3840 avail_mem=1.51 GB):   4%|▍         | 2/50 [00:00<00:03, 15.09it/s]
Capturing prefill shape (num_tokens=3584 avail_mem=1.42 GB):   4%|▍         | 2/50 [00:00<00:03, 15.09it/s]
Capturing prefill shape (num_tokens=3328 avail_mem=1.42 GB):   4%|▍         | 2/50 [00:00<00:03, 15.09it/s]
Capturing prefill shape (num_tokens=3328 avail_mem=1.42 GB):   8%|▊         | 4/50 [00:00<00:03, 12.23it/s]
Capturing prefill shape (num_tokens=3072 avail_mem=1.42 GB):   8%|▊         | 4/50 [00:00<00:03, 12.23it/s]
Capturing prefill shape (num_tokens=2816 avail_mem=1.42 GB):   8%|▊         | 4/50 [00:00<00:03, 12.23it/s]
Capturing prefill shape (num_tokens=2816 avail_mem=1.42 GB):  12%|█▏        | 6/50 [00:00<00:03, 11.94it/s]
Capturing prefill shape (num_tokens=2560 avail_mem=1.42 GB):  12%|█▏        | 6/50 [00:00<00:03, 11.94it/s]
Capturing prefill shape (num_tokens=2304 avail_mem=1.42 GB):  12%|█▏        | 6/50 [00:00<00:03, 11.94it/s]
Capturing prefill shape (num_tokens=2304 avail_mem=1.42 GB):  16%|█▌        | 8/50 [00:00<00:03, 12.70it/s]
Capturing prefill shape (num_tokens=2048 avail_mem=1.42 GB):  16%|█▌        | 8/50 [00:00<00:03, 12.70it/s]
Capturing prefill shape (num_tokens=1792 avail_mem=1.42 GB):  16%|█▌        | 8/50 [00:00<00:03, 12.70it/s]
Capturing prefill shape (num_tokens=1792 avail_mem=1.42 GB):  20%|██        | 10/50 [00:00<00:02, 14.12it/s]
Capturing prefill shape (num_tokens=1536 avail_mem=1.42 GB):  20%|██        | 10/50 [00:00<00:02, 14.12it/s]
Capturing prefill shape (num_tokens=1280 avail_mem=1.42 GB):  20%|██        | 10/50 [00:00<00:02, 14.12it/s]
Capturing prefill shape (num_tokens=1024 avail_mem=1.42 GB):  20%|██        | 10/50 [00:00<00:02, 14.12it/s]
Capturing prefill shape (num_tokens=1024 avail_mem=1.42 GB):  26%|██▌       | 13/50 [00:00<00:02, 17.58it/s]
Capturing prefill shape (num_tokens=960 avail_mem=1.42 GB):  26%|██▌       | 13/50 [00:00<00:02, 17.58it/s] 
Capturing prefill shape (num_tokens=896 avail_mem=1.42 GB):  26%|██▌       | 13/50 [00:00<00:02, 17.58it/s]
Capturing prefill shape (num_tokens=832 avail_mem=1.42 GB):  26%|██▌       | 13/50 [00:00<00:02, 17.58it/s]
Capturing prefill shape (num_tokens=768 avail_mem=1.42 GB):  26%|██▌       | 13/50 [00:00<00:02, 17.58it/s]
Capturing prefill shape (num_tokens=768 avail_mem=1.42 GB):  34%|███▍      | 17/50 [00:00<00:01, 23.46it/s]
Capturing prefill shape (num_tokens=704 avail_mem=1.42 GB):  34%|███▍      | 17/50 [00:00<00:01, 23.46it/s]
Capturing prefill shape (num_tokens=640 avail_mem=1.42 GB):  34%|███▍      | 17/50 [00:00<00:01, 23.46it/s]
Capturing prefill shape (num_tokens=576 avail_mem=1.42 GB):  34%|███▍      | 17/50 [00:01<00:01, 23.46it/s]
Capturing prefill shape (num_tokens=512 avail_mem=1.42 GB):  34%|███▍      | 17/50 [00:01<00:01, 23.46it/s]
Capturing prefill shape (num_tokens=480 avail_mem=1.41 GB):  34%|███▍      | 17/50 [00:01<00:01, 23.46it/s]
Capturing prefill shape (num_tokens=448 avail_mem=1.41 GB):  34%|███▍      | 17/50 [00:01<00:01, 23.46it/s]
Capturing prefill shape (num_tokens=448 avail_mem=1.41 GB):  46%|████▌     | 23/50 [00:01<00:00, 32.81it/s]
Capturing prefill shape (num_tokens=416 avail_mem=1.41 GB):  46%|████▌     | 23/50 [00:01<00:00, 32.81it/s]
Capturing prefill shape (num_tokens=384 avail_mem=1.41 GB):  46%|████▌     | 23/50 [00:01<00:00, 32.81it/s]
Capturing prefill shape (num_tokens=352 avail_mem=1.41 GB):  46%|████▌     | 23/50 [00:01<00:00, 32.81it/s]
Capturing prefill shape (num_tokens=320 avail_mem=1.41 GB):  46%|████▌     | 23/50 [00:01<00:00, 32.81it/s]
Capturing prefill shape (num_tokens=288 avail_mem=1.41 GB):  46%|████▌     | 23/50 [00:01<00:00, 32.81it/s]
Capturing prefill shape (num_tokens=256 avail_mem=1.41 GB):  46%|████▌     | 23/50 [00:01<00:00, 32.81it/s]
Capturing prefill shape (num_tokens=240 avail_mem=1.41 GB):  46%|████▌     | 23/50 [00:01<00:00, 32.81it/s]
Capturing prefill shape (num_tokens=224 avail_mem=1.41 GB):  46%|████▌     | 23/50 [00:01<00:00, 32.81it/s]
Capturing prefill shape (num_tokens=208 avail_mem=1.41 GB):  46%|████▌     | 23/50 [00:01<00:00, 32.81it/s]
Capturing prefill shape (num_tokens=192 avail_mem=1.41 GB):  46%|████▌     | 23/50 [00:01<00:00, 32.81it/s]
Capturing prefill shape (num_tokens=192 avail_mem=1.41 GB):  66%|██████▌   | 33/50 [00:01<00:00, 50.41it/s]
Capturing prefill shape (num_tokens=176 avail_mem=1.41 GB):  66%|██████▌   | 33/50 [00:01<00:00, 50.41it/s]
Capturing prefill shape (num_tokens=160 avail_mem=1.41 GB):  66%|██████▌   | 33/50 [00:01<00:00, 50.41it/s]
Capturing prefill shape (num_tokens=144 avail_mem=1.41 GB):  66%|██████▌   | 33/50 [00:01<00:00, 50.41it/s]
Capturing prefill shape (num_tokens=128 avail_mem=1.41 GB):  66%|██████▌   | 33/50 [00:01<00:00, 50.41it/s]
Capturing prefill shape (num_tokens=112 avail_mem=1.40 GB):  66%|██████▌   | 33/50 [00:01<00:00, 50.41it/s]
Capturing prefill shape (num_tokens=96 avail_mem=1.40 GB):  66%|██████▌   | 33/50 [00:01<00:00, 50.41it/s] 
Capturing prefill shape (num_tokens=80 avail_mem=1.40 GB):  66%|██████▌   | 33/50 [00:01<00:00, 50.41it/s]
Capturing prefill shape (num_tokens=64 avail_mem=1.40 GB):  66%|██████▌   | 33/50 [00:01<00:00, 50.41it/s]
Capturing prefill shape (num_tokens=48 avail_mem=1.40 GB):  66%|██████▌   | 33/50 [00:01<00:00, 50.41it/s]
Capturing prefill shape (num_tokens=32 avail_mem=1.40 GB):  66%|██████▌   | 33/50 [00:01<00:00, 50.41it/s]
Capturing prefill shape (num_tokens=28 avail_mem=1.40 GB):  66%|██████▌   | 33/50 [00:01<00:00, 50.41it/s]
Capturing prefill shape (num_tokens=24 avail_mem=1.40 GB):  66%|██████▌   | 33/50 [00:01<00:00, 50.41it/s]
Capturing prefill shape (num_tokens=20 avail_mem=1.40 GB):  66%|██████▌   | 33/50 [00:01<00:00, 50.41it/s]
Capturing prefill shape (num_tokens=16 avail_mem=1.40 GB):  66%|██████▌   | 33/50 [00:01<00:00, 50.41it/s]
Capturing prefill shape (num_tokens=12 avail_mem=1.40 GB):  66%|██████▌   | 33/50 [00:01<00:00, 50.41it/s]
Capturing prefill shape (num_tokens=12 avail_mem=1.40 GB):  96%|█████████▌| 48/50 [00:01<00:00, 77.70it/s]
Capturing prefill shape (num_tokens=8 avail_mem=1.40 GB):  96%|█████████▌| 48/50 [00:01<00:00, 77.70it/s] 
Capturing prefill shape (num_tokens=4 avail_mem=1.40 GB):  96%|█████████▌| 48/50 [00:01<00:00, 77.70it/s]
Capturing prefill shape (num_tokens=4 avail_mem=1.40 GB): 100%|██████████| 50/50 [00:01<00:00, 38.63it/s]
[2026-10-17 15:29:18] Capture draft prefill CUDA graph end. elapsed=2.10 s, mem usage=0.21 GB, avail mem=1.98 GB.
[2026-10-17 15:29:18] Capture draft decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 40, 44, 48], avail mem=1.98 GB

  0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=48 avail_mem=1.95 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=48 avail_mem=1.95 GB):   4%|▍         | 1/23 [00:00<00:20,  1.08it/s]
Capturing batches (bs=44 avail_mem=1.93 GB):   4%|▍         | 1/23 [00:00<00:20,  1.08it/s]
Capturing batches (bs=40 avail_mem=1.91 GB):   4%|▍         | 1/23 [00:00<00:20,  1.08it/s]
Capturing batches (bs=32 avail_mem=1.89 GB):   4%|▍         | 1/23 [00:01<00:20,  1.08it/s]
Capturing batches (bs=32 avail_mem=1.89 GB):  17%|█▋        | 4/23 [00:01<00:03,  4.82it/s]
Capturing batches (bs=30 avail_mem=1.87 GB):  17%|█▋        | 4/23 [00:01<00:03,  4.82it/s]
Capturing batches (bs=28 avail_mem=1.85 GB):  17%|█▋        | 4/23 [00:01<00:03,  4.82it/s]
Capturing batches (bs=26 avail_mem=1.83 GB):  17%|█▋        | 4/23 [00:01<00:03,  4.82it/s]
Capturing batches (bs=26 avail_mem=1.83 GB):  30%|███       | 7/23 [00:01<00:01,  8.53it/s]
Capturing batches (bs=24 avail_mem=1.81 GB):  30%|███       | 7/23 [00:01<00:01,  8.53it/s]
Capturing batches (bs=22 avail_mem=1.79 GB):  30%|███       | 7/23 [00:01<00:01,  8.53it/s]
Capturing batches (bs=20 avail_mem=1.77 GB):  30%|███       | 7/23 [00:01<00:01,  8.53it/s]
Capturing batches (bs=20 avail_mem=1.77 GB):  43%|████▎     | 10/23 [00:01<00:01, 12.15it/s]
Capturing batches (bs=18 avail_mem=1.75 GB):  43%|████▎     | 10/23 [00:01<00:01, 12.15it/s]
Capturing batches (bs=16 avail_mem=1.73 GB):  43%|████▎     | 10/23 [00:01<00:01, 12.15it/s]
Capturing batches (bs=14 avail_mem=1.71 GB):  43%|████▎     | 10/23 [00:01<00:01, 12.15it/s]
Capturing batches (bs=14 avail_mem=1.71 GB):  57%|█████▋    | 13/23 [00:01<00:00, 15.51it/s]
Capturing batches (bs=12 avail_mem=1.69 GB):  57%|█████▋    | 13/23 [00:01<00:00, 15.51it/s]
Capturing batches (bs=10 avail_mem=1.67 GB):  57%|█████▋    | 13/23 [00:01<00:00, 15.51it/s]
Capturing batches (bs=8 avail_mem=1.65 GB):  57%|█████▋    | 13/23 [00:01<00:00, 15.51it/s] 
Capturing batches (bs=8 avail_mem=1.65 GB):  70%|██████▉   | 16/23 [00:01<00:00, 18.49it/s]
Capturing batches (bs=7 avail_mem=1.63 GB):  70%|██████▉   | 16/23 [00:01<00:00, 18.49it/s]
Capturing batches (bs=6 avail_mem=1.61 GB):  70%|██████▉   | 16/23 [00:01<00:00, 18.49it/s]
Capturing batches (bs=5 avail_mem=1.59 GB):  70%|██████▉   | 16/23 [00:01<00:00, 18.49it/s]
Capturing batches (bs=5 avail_mem=1.59 GB):  83%|████████▎ | 19/23 [00:01<00:00, 21.05it/s]
Capturing batches (bs=4 avail_mem=1.57 GB):  83%|████████▎ | 19/23 [00:01<00:00, 21.05it/s]
Capturing batches (bs=3 avail_mem=1.55 GB):  83%|████████▎ | 19/23 [00:01<00:00, 21.05it/s]
Capturing batches (bs=2 avail_mem=1.53 GB):  83%|████████▎ | 19/23 [00:01<00:00, 21.05it/s]
Capturing batches (bs=2 avail_mem=1.53 GB):  96%|█████████▌| 22/23 [00:01<00:00, 22.88it/s]
Capturing batches (bs=1 avail_mem=1.51 GB):  96%|█████████▌| 22/23 [00:01<00:00, 22.88it/s]
Capturing batches (bs=1 avail_mem=1.51 GB): 100%|██████████| 23/23 [00:01<00:00, 11.65it/s]
[2026-10-17 15:29:21] Capture draft decode CUDA graph end. elapsed=2.79 s, mem usage=0.49 GB, avail mem=1.49 GB.
[2026-10-17 15:29:21] Capture draft extend CUDA graph begin. backend=full, num_tokens_per_req=4, bs=[1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 40, 44, 48], avail mem=1.49 GB

  0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=48 avail_mem=1.44 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=44 avail_mem=1.42 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=40 avail_mem=1.42 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=32 avail_mem=1.40 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=30 avail_mem=1.40 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=30 avail_mem=1.40 GB):  22%|██▏       | 5/23 [00:00<00:00, 45.82it/s]
Capturing batches (bs=28 avail_mem=1.38 GB):  22%|██▏       | 5/23 [00:00<00:00, 45.82it/s]
Capturing batches (bs=26 avail_mem=1.37 GB):  22%|██▏       | 5/23 [00:00<00:00, 45.82it/s]
Capturing batches (bs=24 avail_mem=1.35 GB):  22%|██▏       | 5/23 [00:00<00:00, 45.82it/s]
Capturing batches (bs=22 avail_mem=1.35 GB):  22%|██▏       | 5/23 [00:00<00:00, 45.82it/s]
Capturing batches (bs=20 avail_mem=1.33 GB):  22%|██▏       | 5/23 [00:00<00:00, 45.82it/s]
Capturing batches (bs=20 avail_mem=1.33 GB):  43%|████▎     | 10/23 [00:00<00:00, 46.55it/s]
Capturing batches (bs=18 avail_mem=1.33 GB):  43%|████▎     | 10/23 [00:00<00:00, 46.55it/s]
Capturing batches (bs=16 avail_mem=1.31 GB):  43%|████▎     | 10/23 [00:00<00:00, 46.55it/s]
Capturing batches (bs=14 avail_mem=1.31 GB):  43%|████▎     | 10/23 [00:00<00:00, 46.55it/s]
Capturing batches (bs=12 avail_mem=1.29 GB):  43%|████▎     | 10/23 [00:00<00:00, 46.55it/s]
Capturing batches (bs=10 avail_mem=1.29 GB):  43%|████▎     | 10/23 [00:00<00:00, 46.55it/s]
Capturing batches (bs=10 avail_mem=1.29 GB):  65%|██████▌   | 15/23 [00:00<00:00, 47.45it/s]
Capturing batches (bs=8 avail_mem=1.27 GB):  65%|██████▌   | 15/23 [00:00<00:00, 47.45it/s] 
Capturing batches (bs=7 avail_mem=1.27 GB):  65%|██████▌   | 15/23 [00:00<00:00, 47.45it/s]
Capturing batches (bs=6 avail_mem=1.25 GB):  65%|██████▌   | 15/23 [00:00<00:00, 47.45it/s]
Capturing batches (bs=5 avail_mem=1.25 GB):  65%|██████▌   | 15/23 [00:00<00:00, 47.45it/s]
Capturing batches (bs=4 avail_mem=1.23 GB):  65%|██████▌   | 15/23 [00:00<00:00, 47.45it/s]
Capturing batches (bs=4 avail_mem=1.23 GB):  87%|████████▋ | 20/23 [00:00<00:00, 46.60it/s]
Capturing batches (bs=3 avail_mem=1.23 GB):  87%|████████▋ | 20/23 [00:00<00:00, 46.60it/s]
Capturing batches (bs=2 avail_mem=1.21 GB):  87%|████████▋ | 20/23 [00:00<00:00, 46.60it/s]
Capturing batches (bs=1 avail_mem=1.21 GB):  87%|████████▋ | 20/23 [00:00<00:00, 46.60it/s]
Capturing batches (bs=1 avail_mem=1.21 GB): 100%|██████████| 23/23 [00:00<00:00, 47.28it/s]
[2026-10-17 15:29:22] Capture draft extend CUDA graph end. elapsed=1.34 s, mem usage=0.30 GB, avail mem=1.19 GB.
[2026-10-17 15:29:22] max_total_num_tokens=8827, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=48, context_len=40960, available_gpu_mem=1.19 GB
[2026-10-17 15:29:22] Init Unified Radix Cache. Components: (<ComponentType.FULL: 0>,). Tree Core: RustUnifiedTreeCore
[2026-10-17 15:29:22] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=False hybrid_ssm=False hicache_attached=False
[2026-10-17 15:29:23] Engine startup timings (s): load_weight=4.35, kv_cache_allocation=0.76, scheduler_e2e=62.74, cuda_graph={prefill=41.43, decode=0.00, target_verify=5.37, draft_prefill=2.10, draft_decode=2.79, draft_extend=1.34}, tokenizer_e2e=72.64
[2026-10-17 15:29:23] INFO:     Started server process [1341742]
[2026-10-17 15:29:23] INFO:     Waiting for application startup.
[2026-10-17 15:29:23] Using default chat sampling params from model generation config: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
[2026-10-17 15:29:23] INFO:     Application startup complete.
[2026-10-17 15:29:23] INFO:     Uvicorn running on http://127.0.0.1:30000 (Press CTRL+C to quit)
[2026-10-17 15:29:24] INFO:     127.0.0.1:59290 - "GET /health HTTP/1.1" 503 Service Unavailable
[2026-10-17 15:29:24] INFO:     127.0.0.1:59304 - "GET /model_info HTTP/1.1" 200 OK
[2026-10-17 15:29:24] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 3.23
[2026-10-17 15:29:25] INFO:     127.0.0.1:59308 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:29:25] INFO:     127.0.0.1:55124 - "POST /freeze_gc HTTP/1.1" 200 OK
[2026-10-17 15:29:25] The server is fired up and ready to roll!
[2026-10-17 15:29:26] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.58
[2026-10-17 15:29:27] INFO:     127.0.0.1:55140 - "GET /health HTTP/1.1" 200 OK
[2026-10-17 15:29:27] Prefill batch, #new-seq: 1, #new-token: 4, #cached-token: 1, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 3.73
[2026-10-17 15:29:27] INFO:     127.0.0.1:55148 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:29:28] Prefill batch, #new-seq: 1, #new-token: 21, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 35.94
[2026-10-17 15:29:28] INFO:     127.0.0.1:55162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
/data/sglang/.venv/lib/python3.12/site-packages/sglang/srt/entrypoints/http_server.py:476: FastAPIDeprecationWarning: ORJSONResponse is deprecated, FastAPI now serializes data directly to JSON bytes via Pydantic when a return type or response model is set, which is faster and doesn't need a custom response class. Read more in the FastAPI docs: https://fastapi.tiangolo.com/advanced/custom-response/#orjson-or-response-model and https://fastapi.tiangolo.com/tutorial/response-model/
  return await original_handler(ORJSONRequest(request.scope, request.receive))
[2026-10-17 15:29:36] INFO:     127.0.0.1:49958 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-17 15:29:36] INFO:     127.0.0.1:49966 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-17 15:29:37] INFO:     127.0.0.1:49968 - "GET /model_info HTTP/1.1" 200 OK
[2026-10-17 15:29:45] INFO:     127.0.0.1:32838 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:29:46] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 28.60
[2026-10-17 15:29:47] INFO:     127.0.0.1:32854 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:29:47] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1.19
[2026-10-17 15:29:48] Decode batch, #running-req: 1, #token: 570, token usage: 0.06, accept len: 2.80, accept rate: 0.60, cuda graph: True, gen throughput (token/s): 4.31, #queue-req: 0
[2026-10-17 15:29:50] INFO:     127.0.0.1:32856 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:29:50] Prefill batch, #new-seq: 1, #new-token: 256, #cached-token: 0, token usage: 0.03, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 84.18
[2026-10-17 15:29:51] Decode batch, #running-req: 1, #token: 279, token usage: 0.03, accept len: 2.33, accept rate: 0.44, cuda graph: True, gen throughput (token/s): 30.60, #queue-req: 0
[2026-10-17 15:29:54] Decode batch, #running-req: 1, #token: 357, token usage: 0.04, accept len: 1.98, accept rate: 0.33, cuda graph: True, gen throughput (token/s): 27.03, #queue-req: 0
[2026-10-17 15:29:56] INFO:     127.0.0.1:44664 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:29:56] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 92.26
[2026-10-17 15:29:57] Decode batch, #running-req: 1, #token: 592, token usage: 0.07, accept len: 2.73, accept rate: 0.57, cuda graph: True, gen throughput (token/s): 34.67, #queue-req: 0
[2026-10-17 15:29:59] INFO:     127.0.0.1:44678 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:29:59] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 157.63
[2026-10-17 15:30:01] Decode batch, #running-req: 1, #token: 542, token usage: 0.06, accept len: 2.00, accept rate: 0.33, cuda graph: True, gen throughput (token/s): 24.55, #queue-req: 0
[2026-10-17 15:30:04] Decode batch, #running-req: 1, #token: 591, token usage: 0.07, accept len: 1.25, accept rate: 0.08, cuda graph: True, gen throughput (token/s): 17.02, #queue-req: 0
[2026-10-17 15:30:05] INFO:     127.0.0.1:56806 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:30:06] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 76.83
[2026-10-17 15:30:07] Decode batch, #running-req: 1, #token: 577, token usage: 0.07, accept len: 2.90, accept rate: 0.63, cuda graph: True, gen throughput (token/s): 36.13, #queue-req: 0
[2026-10-17 15:30:08] INFO:     127.0.0.1:56812 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:30:09] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 171.12
[2026-10-17 15:30:10] Decode batch, #running-req: 1, #token: 588, token usage: 0.07, accept len: 3.60, accept rate: 0.87, cuda graph: True, gen throughput (token/s): 41.32, #queue-req: 0
[2026-10-17 15:30:12] INFO:     127.0.0.1:56816 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:30:12] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 140.54
[2026-10-17 15:30:14] Decode batch, #running-req: 1, #token: 559, token usage: 0.06, accept len: 2.45, accept rate: 0.48, cuda graph: True, gen throughput (token/s): 27.01, #queue-req: 0
[2026-10-17 15:30:17] Decode batch, #running-req: 1, #token: 638, token usage: 0.07, accept len: 2.00, accept rate: 0.33, cuda graph: True, gen throughput (token/s): 25.60, #queue-req: 0
[2026-10-17 15:30:17] INFO:     127.0.0.1:39010 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:30:18] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 91.85
[2026-10-17 15:30:21] INFO:     127.0.0.1:39026 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:30:21] Prefill batch, #new-seq: 1, #new-token: 497, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 152.88
[2026-10-17 15:30:21] Decode batch, #running-req: 1, #token: 505, token usage: 0.06, accept len: 3.65, accept rate: 0.88, cuda graph: True, gen throughput (token/s): 35.39, #queue-req: 0
[2026-10-17 15:30:24] INFO:     127.0.0.1:39042 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:30:24] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 158.17
[2026-10-17 15:30:25] Decode batch, #running-req: 1, #token: 537, token usage: 0.06, accept len: 3.70, accept rate: 0.90, cuda graph: True, gen throughput (token/s): 40.75, #queue-req: 0
[2026-10-17 15:30:28] Decode batch, #running-req: 1, #token: 644, token usage: 0.07, accept len: 2.60, accept rate: 0.53, cuda graph: True, gen throughput (token/s): 33.27, #queue-req: 0
[2026-10-17 15:30:28] INFO:     127.0.0.1:39314 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:30:29] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 122.70
[2026-10-17 15:30:31] INFO:     127.0.0.1:39316 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:30:32] Prefill batch, #new-seq: 1, #new-token: 509, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 165.03
[2026-10-17 15:30:32] Decode batch, #running-req: 1, #token: 532, token usage: 0.06, accept len: 3.77, accept rate: 0.93, cuda graph: True, gen throughput (token/s): 36.58, #queue-req: 0
[2026-10-17 15:30:35] INFO:     127.0.0.1:34016 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:30:35] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 147.47
[2026-10-17 15:30:36] Decode batch, #running-req: 1, #token: 543, token usage: 0.06, accept len: 3.48, accept rate: 0.82, cuda graph: True, gen throughput (token/s): 38.35, #queue-req: 0
[2026-10-17 15:30:38] INFO:     127.0.0.1:34030 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:30:39] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 141.42
[2026-10-17 15:30:39] Decode batch, #running-req: 1, #token: 535, token usage: 0.06, accept len: 3.00, accept rate: 0.67, cuda graph: True, gen throughput (token/s): 33.13, #queue-req: 0
[2026-10-17 15:30:43] Decode batch, #running-req: 1, #token: 603, token usage: 0.07, accept len: 1.70, accept rate: 0.23, cuda graph: True, gen throughput (token/s): 21.65, #queue-req: 0
[2026-10-17 15:30:44] INFO:     127.0.0.1:34042 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:30:44] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 89.25
[2026-10-17 15:30:47] Decode batch, #running-req: 1, #token: 604, token usage: 0.07, accept len: 3.30, accept rate: 0.77, cuda graph: True, gen throughput (token/s): 27.17, #queue-req: 0
[2026-10-17 15:30:49] INFO:     127.0.0.1:51906 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:30:50] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 99.80
[2026-10-17 15:30:52] Decode batch, #running-req: 1, #token: 620, token usage: 0.07, accept len: 3.73, accept rate: 0.91, cuda graph: True, gen throughput (token/s): 31.13, #queue-req: 0
[2026-10-17 15:30:53] INFO:     127.0.0.1:51916 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-17 15:30:53] INFO:     127.0.0.1:51924 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-17 15:31:03] INFO:     127.0.0.1:34426 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-17 15:31:03] INFO:     127.0.0.1:34434 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-17 15:31:04] INFO:     127.0.0.1:34438 - "GET /model_info HTTP/1.1" 200 OK
[2026-10-17 15:31:12] INFO:     127.0.0.1:38138 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:12] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 22.81
[2026-10-17 15:31:14] INFO:     127.0.0.1:38148 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:14] INFO:     127.0.0.1:38162 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:14] INFO:     127.0.0.1:38178 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:14] INFO:     127.0.0.1:38184 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:14] INFO:     127.0.0.1:38188 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:14] INFO:     127.0.0.1:38192 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:14] INFO:     127.0.0.1:38204 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:14] INFO:     127.0.0.1:38206 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:14] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.44, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1.03
[2026-10-17 15:31:14] Prefill batch, #new-seq: 7, #new-token: 778, #cached-token: 2550, token usage: 0.44, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 7799.56
[2026-10-17 15:31:16] Decode batch, #running-req: 8, #token: 4418, token usage: 0.50, accept len: 2.88, accept rate: 0.62, cuda graph: True, gen throughput (token/s): 25.01, #queue-req: 0
[2026-10-17 15:31:17] INFO:     127.0.0.1:37528 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:17] INFO:     127.0.0.1:37538 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:18] Prefill batch, #new-seq: 2, #new-token: 1008, #cached-token: 0, token usage: 0.50, #running-req: 6, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 291.62
[2026-10-17 15:31:18] INFO:     127.0.0.1:37544 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:18] INFO:     127.0.0.1:37556 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:18] INFO:     127.0.0.1:37566 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:19] Prefill batch, #new-seq: 2, #new-token: 1021, #cached-token: 0, token usage: 0.46, #running-req: 5, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 656.79
[2026-10-17 15:31:19] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.47, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 22111.52
[2026-10-17 15:31:21] INFO:     127.0.0.1:37576 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:21] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.51, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 248.80
[2026-10-17 15:31:21] Decode batch, #running-req: 8, #token: 4540, token usage: 0.51, accept len: 2.96, accept rate: 0.65, cuda graph: True, gen throughput (token/s): 171.70, #queue-req: 0
[2026-10-17 15:31:22] INFO:     127.0.0.1:37592 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:22] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.51, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 461.86
[2026-10-17 15:31:23] INFO:     127.0.0.1:37598 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:23] INFO:     127.0.0.1:37608 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:24] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.52, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 383.49
[2026-10-17 15:31:24] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.53, #running-req: 8, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 19154.66
[2026-10-17 15:31:24] INFO:     127.0.0.1:37622 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:24] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.52, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 897.40
[2026-10-17 15:31:24] INFO:     127.0.0.1:35936 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:25] INFO:     127.0.0.1:35948 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:25] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.44, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 795.77
[2026-10-17 15:31:25] Prefill batch, #new-seq: 1, #new-token: 507, #cached-token: 0, token usage: 0.50, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1312.42
[2026-10-17 15:31:26] INFO:     127.0.0.1:35958 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:26] Prefill batch, #new-seq: 1, #new-token: 488, #cached-token: 0, token usage: 0.50, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 598.38
[2026-10-17 15:31:28] INFO:     127.0.0.1:35966 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:28] Decode batch, #running-req: 7, #token: 4626, token usage: 0.52, accept len: 2.93, accept rate: 0.64, cuda graph: True, gen throughput (token/s): 131.11, #queue-req: 0
[2026-10-17 15:31:28] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.53, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 217.97
[2026-10-17 15:31:29] INFO:     127.0.0.1:35982 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:30] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.53, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 458.20
[2026-10-17 15:31:30] INFO:     127.0.0.1:35998 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:31] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.54, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 360.29
[2026-10-17 15:31:31] INFO:     127.0.0.1:36014 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:32] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.53, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 657.57
[2026-10-17 15:31:32] INFO:     127.0.0.1:36028 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:33] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.52, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 568.59
[2026-10-17 15:31:33] INFO:     127.0.0.1:36032 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:34] INFO:     127.0.0.1:36034 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:34] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.45, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 469.94
[2026-10-17 15:31:34] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.51, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1210.46
[2026-10-17 15:31:35] INFO:     127.0.0.1:57314 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:35] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.52, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 410.52
[2026-10-17 15:31:36] Decode batch, #running-req: 8, #token: 4656, token usage: 0.53, accept len: 2.87, accept rate: 0.62, cuda graph: True, gen throughput (token/s): 120.62, #queue-req: 0
[2026-10-17 15:31:36] INFO:     127.0.0.1:57316 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:36] INFO:     127.0.0.1:57330 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:37] Prefill batch, #new-seq: 2, #new-token: 1024, #cached-token: 0, token usage: 0.51, #running-req: 6, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 691.25
[2026-10-17 15:31:38] INFO:     127.0.0.1:57346 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:38] Prefill batch, #new-seq: 1, #new-token: 510, #cached-token: 0, token usage: 0.52, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 424.29
[2026-10-17 15:31:39] INFO:     127.0.0.1:57350 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:39] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.53, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 418.35
[2026-10-17 15:31:40] INFO:     127.0.0.1:57366 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:40] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.53, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 670.89
[2026-10-17 15:31:40] INFO:     127.0.0.1:57374 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:41] Prefill batch, #new-seq: 1, #new-token: 508, #cached-token: 0, token usage: 0.53, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 533.57
[2026-10-17 15:31:41] INFO:     127.0.0.1:57378 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:42] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.52, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 731.62
[2026-10-17 15:31:42] INFO:     127.0.0.1:57386 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:43] Decode batch, #running-req: 7, #token: 4541, token usage: 0.51, accept len: 3.06, accept rate: 0.69, cuda graph: True, gen throughput (token/s): 140.36, #queue-req: 0
[2026-10-17 15:31:43] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.52, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 580.01
[2026-10-17 15:31:43] INFO:     127.0.0.1:57398 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:44] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.53, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 429.51
[2026-10-17 15:31:44] INFO:     127.0.0.1:57408 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:45] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.53, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 514.84
[2026-10-17 15:31:45] INFO:     127.0.0.1:34706 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:46] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.53, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 516.09
[2026-10-17 15:31:46] INFO:     127.0.0.1:34710 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:46] INFO:     127.0.0.1:34718 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:47] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.51, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 338.98
[2026-10-17 15:31:47] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.51, #running-req: 8, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 19024.76
[2026-10-17 15:31:48] INFO:     127.0.0.1:34724 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:48] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.51, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 624.56
[2026-10-17 15:31:49] Decode batch, #running-req: 8, #token: 4662, token usage: 0.53, accept len: 2.80, accept rate: 0.60, cuda graph: True, gen throughput (token/s): 135.53, #queue-req: 0
[2026-10-17 15:31:50] INFO:     127.0.0.1:34730 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:50] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.54, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 232.76
[2026-10-17 15:31:50] INFO:     127.0.0.1:34734 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:51] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.53, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 710.81
[2026-10-17 15:31:51] INFO:     127.0.0.1:34746 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:52] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.53, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 535.55
[2026-10-17 15:31:52] INFO:     127.0.0.1:34748 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:52] INFO:     127.0.0.1:34758 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:54] Prefill batch, #new-seq: 2, #new-token: 1242, #cached-token: 0, token usage: 0.53, #running-req: 6, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 598.28
[2026-10-17 15:31:55] INFO:     127.0.0.1:37508 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:55] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.55, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 401.41
[2026-10-17 15:31:55] INFO:     127.0.0.1:37520 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:56] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.54, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1007.13
[2026-10-17 15:31:56] Decode batch, #running-req: 8, #token: 4209, token usage: 0.48, accept len: 2.85, accept rate: 0.62, cuda graph: True, gen throughput (token/s): 121.44, #queue-req: 0
[2026-10-17 15:31:56] INFO:     127.0.0.1:37522 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:57] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.54, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 439.29
[2026-10-17 15:31:58] INFO:     127.0.0.1:37532 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:31:59] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.56, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 299.48
[2026-10-17 15:31:59] INFO:     127.0.0.1:37536 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:00] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.55, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 461.48
[2026-10-17 15:32:00] INFO:     127.0.0.1:37546 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:01] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.53, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 412.72
[2026-10-17 15:32:02] INFO:     127.0.0.1:37548 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:03] Prefill batch, #new-seq: 1, #new-token: 510, #cached-token: 0, token usage: 0.54, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 325.87
[2026-10-17 15:32:03] INFO:     127.0.0.1:37554 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:03] INFO:     127.0.0.1:37570 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:04] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.51, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 365.33
[2026-10-17 15:32:04] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.51, #running-req: 8, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 16232.05
[2026-10-17 15:32:04] INFO:     127.0.0.1:34918 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:05] Decode batch, #running-req: 7, #token: 3798, token usage: 0.43, accept len: 2.87, accept rate: 0.62, cuda graph: True, gen throughput (token/s): 100.01, #queue-req: 0
[2026-10-17 15:32:05] INFO:     127.0.0.1:34926 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:05] Prefill batch, #new-seq: 1, #new-token: 493, #cached-token: 0, token usage: 0.44, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 445.51
[2026-10-17 15:32:06] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.50, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 799.32
[2026-10-17 15:32:08] INFO:     127.0.0.1:34928 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:08] INFO:     127.0.0.1:34936 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:10] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.54, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 138.07
[2026-10-17 15:32:10] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.55, #running-req: 8, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 15210.11
[2026-10-17 15:32:10] INFO:     127.0.0.1:34946 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:10] INFO:     127.0.0.1:34958 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:10] INFO:     127.0.0.1:34966 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:12] Prefill batch, #new-seq: 2, #new-token: 854, #cached-token: 0, token usage: 0.42, #running-req: 6, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 402.66
[2026-10-17 15:32:12] Prefill batch, #new-seq: 1, #new-token: 509, #cached-token: 0, token usage: 0.42, #running-req: 8, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 19687.96
[2026-10-17 15:32:12] Decode batch, #running-req: 6, #token: 3150, token usage: 0.36, accept len: 2.93, accept rate: 0.64, cuda graph: True, gen throughput (token/s): 130.57, #queue-req: 0
[2026-10-17 15:32:15] Decode batch, #running-req: 1, #token: 602, token usage: 0.07, accept len: 3.11, accept rate: 0.70, cuda graph: True, gen throughput (token/s): 158.20, #queue-req: 0
[2026-10-17 15:32:17] INFO:     127.0.0.1:44176 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-17 15:32:17] INFO:     127.0.0.1:44190 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-17 15:32:28] INFO:     127.0.0.1:36188 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-17 15:32:28] INFO:     127.0.0.1:36196 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-17 15:32:28] INFO:     127.0.0.1:36208 - "GET /model_info HTTP/1.1" 200 OK
[2026-10-17 15:32:36] INFO:     127.0.0.1:58210 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:36] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.06, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 20.78
[2026-10-17 15:32:38] INFO:     127.0.0.1:58212 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58228 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58236 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58244 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58246 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58260 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58268 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58278 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58286 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58298 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58302 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58310 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58314 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58316 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58324 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58338 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58352 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58368 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58384 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58392 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58402 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58404 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58418 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58434 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58446 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58454 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58468 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58476 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58480 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58484 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58494 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:38] INFO:     127.0.0.1:58504 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:40] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.52, #running-req: 0, #queue-req: 11, #pending-token: 0, cuda graph: True, input throughput (token/s): 0.61
[2026-10-17 15:32:41] Prefill batch, #new-seq: 9, #new-token: 4096, #cached-token: 0, token usage: 0.78, #running-req: 1, #queue-req: 18, #pending-token: 5840, cuda graph: True, input throughput (token/s): 3677.64
[2026-10-17 15:32:41] Triton kernel 'assign_req_to_token_pool' device-loaded after serving started (free device mem: 0.85 GiB). Pre-load it during engine init to avoid CUDA OOM.
[2026-10-17 15:32:41] Prefill batch, #new-seq: 5, #new-token: 2283, #cached-token: 2, token usage: 0.79, #running-req: 9, #queue-req: 18, #pending-token: 9183, cuda graph: True, input throughput (token/s): 47225.03
[2026-10-17 15:32:41] Triton kernel 'assign_req_to_token_pool' device-loaded after serving started (free device mem: 0.85 GiB). Pre-load it during engine init to avoid CUDA OOM.
[2026-10-17 15:32:42] Decode batch, #running-req: 14, #token: 7426, token usage: 0.84, accept len: 2.90, accept rate: 0.63, cuda graph: True, gen throughput (token/s): 20.06, #queue-req: 18
[2026-10-17 15:32:44] INFO:     127.0.0.1:58512 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:44] INFO:     127.0.0.1:58526 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:44] INFO:     127.0.0.1:58538 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:44] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.79, #running-req: 13, #queue-req: 18, #pending-token: 8671, cuda graph: True, input throughput (token/s): 153.71
[2026-10-17 15:32:45] INFO:     127.0.0.1:34564 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:45] Prefill batch, #new-seq: 2, #new-token: 1024, #cached-token: 0, token usage: 0.84, #running-req: 12, #queue-req: 18, #pending-token: 9178, cuda graph: True, input throughput (token/s): 1238.13
[2026-10-17 15:32:45] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.90, #running-req: 13, #queue-req: 18, #pending-token: 9178, cuda graph: True, input throughput (token/s): 1288.53
[2026-10-17 15:32:45] INFO:     127.0.0.1:34570 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:46] INFO:     127.0.0.1:34578 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:46] INFO:     127.0.0.1:34592 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:46] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.68, #running-req: 13, #queue-req: 18, #pending-token: 9178, cuda graph: True, input throughput (token/s): 1106.46
[2026-10-17 15:32:46] INFO:     127.0.0.1:34602 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:47] INFO:     127.0.0.1:34608 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:47] Prefill batch, #new-seq: 3, #new-token: 1507, #cached-token: 0, token usage: 0.78, #running-req: 11, #queue-req: 18, #pending-token: 8183, cuda graph: True, input throughput (token/s): 1541.95
[2026-10-17 15:32:47] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.85, #running-req: 13, #queue-req: 18, #pending-token: 9206, cuda graph: True, input throughput (token/s): 1071.61
[2026-10-17 15:32:48] INFO:     127.0.0.1:34618 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:48] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.87, #running-req: 13, #queue-req: 18, #pending-token: 8694, cuda graph: True, input throughput (token/s): 422.05
[2026-10-17 15:32:49] Decode batch, #running-req: 14, #token: 7725, token usage: 0.88, accept len: 2.91, accept rate: 0.64, cuda graph: True, gen throughput (token/s): 230.88, #queue-req: 18
[2026-10-17 15:32:50] INFO:     127.0.0.1:34628 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:50] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.93, #running-req: 13, #queue-req: 18, #pending-token: 9205, cuda graph: True, input throughput (token/s): 255.80
[2026-10-17 15:32:51] INFO:     127.0.0.1:34632 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:51] INFO:     127.0.0.1:34636 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:51] INFO:     127.0.0.1:34644 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:52] Prefill batch, #new-seq: 2, #new-token: 1023, #cached-token: 0, token usage: 0.84, #running-req: 12, #queue-req: 18, #pending-token: 8694, cuda graph: True, input throughput (token/s): 989.98
[2026-10-17 15:32:52] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.90, #running-req: 13, #queue-req: 18, #pending-token: 9205, cuda graph: True, input throughput (token/s): 1051.80
[2026-10-17 15:32:52] INFO:     127.0.0.1:34660 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:53] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.90, #running-req: 13, #queue-req: 18, #pending-token: 8694, cuda graph: True, input throughput (token/s): 873.27
[2026-10-17 15:32:53] INFO:     127.0.0.1:34670 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:53] INFO:     127.0.0.1:34680 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:53] INFO:     127.0.0.1:34694 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:54] Prefill batch, #new-seq: 2, #new-token: 1024, #cached-token: 0, token usage: 0.85, #running-req: 12, #queue-req: 19, #pending-token: 9424, cuda graph: True, input throughput (token/s): 702.50
[2026-10-17 15:32:54] INFO:     127.0.0.1:34702 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:55] Prefill batch, #new-seq: 1, #new-token: 510, #cached-token: 0, token usage: 0.85, #running-req: 12, #queue-req: 19, #pending-token: 9426, cuda graph: True, input throughput (token/s): 885.23
[2026-10-17 15:32:55] INFO:     127.0.0.1:55402 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:55] INFO:     127.0.0.1:55414 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:55] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.78, #running-req: 12, #queue-req: 19, #pending-token: 9938, cuda graph: True, input throughput (token/s): 656.95
[2026-10-17 15:32:56] INFO:     127.0.0.1:55428 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:56] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.77, #running-req: 12, #queue-req: 19, #pending-token: 9937, cuda graph: True, input throughput (token/s): 1057.49
[2026-10-17 15:32:56] Prefill batch, #new-seq: 1, #new-token: 508, #cached-token: 0, token usage: 0.83, #running-req: 12, #queue-req: 19, #pending-token: 9941, cuda graph: True, input throughput (token/s): 1067.09
[2026-10-17 15:32:57] Decode batch, #running-req: 13, #token: 7474, token usage: 0.85, accept len: 2.92, accept rate: 0.64, cuda graph: True, gen throughput (token/s): 190.09, #queue-req: 19
[2026-10-17 15:32:57] INFO:     127.0.0.1:55444 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:57] INFO:     127.0.0.1:55452 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:58] Prefill batch, #new-seq: 2, #new-token: 1024, #cached-token: 0, token usage: 0.85, #running-req: 12, #queue-req: 19, #pending-token: 8917, cuda graph: True, input throughput (token/s): 553.99
[2026-10-17 15:32:59] INFO:     127.0.0.1:55468 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:32:59] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.87, #running-req: 12, #queue-req: 19, #pending-token: 9426, cuda graph: True, input throughput (token/s): 424.49
[2026-10-17 15:33:00] INFO:     127.0.0.1:55478 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:00] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.87, #running-req: 12, #queue-req: 19, #pending-token: 9918, cuda graph: True, input throughput (token/s): 741.84
[2026-10-17 15:33:00] INFO:     127.0.0.1:55482 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:01] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.87, #running-req: 12, #queue-req: 19, #pending-token: 9919, cuda graph: True, input throughput (token/s): 648.50
[2026-10-17 15:33:01] INFO:     127.0.0.1:55496 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:02] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.86, #running-req: 12, #queue-req: 19, #pending-token: 9919, cuda graph: True, input throughput (token/s): 743.66
[2026-10-17 15:33:02] INFO:     127.0.0.1:55500 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:02] INFO:     127.0.0.1:55502 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:03] INFO:     127.0.0.1:55508 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:03] Prefill batch, #new-seq: 2, #new-token: 1023, #cached-token: 0, token usage: 0.78, #running-req: 11, #queue-req: 19, #pending-token: 9408, cuda graph: True, input throughput (token/s): 891.32
[2026-10-17 15:33:03] INFO:     127.0.0.1:55512 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:03] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.77, #running-req: 12, #queue-req: 19, #pending-token: 9750, cuda graph: True, input throughput (token/s): 1076.66
[2026-10-17 15:33:04] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.83, #running-req: 12, #queue-req: 19, #pending-token: 9747, cuda graph: True, input throughput (token/s): 1065.57
[2026-10-17 15:33:04] INFO:     127.0.0.1:57902 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:05] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.85, #running-req: 12, #queue-req: 19, #pending-token: 9748, cuda graph: True, input throughput (token/s): 394.29
[2026-10-17 15:33:05] Decode batch, #running-req: 13, #token: 6850, token usage: 0.78, accept len: 2.83, accept rate: 0.61, cuda graph: True, gen throughput (token/s): 175.32, #queue-req: 19
[2026-10-17 15:33:05] INFO:     127.0.0.1:57906 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:06] Prefill batch, #new-seq: 1, #new-token: 731, #cached-token: 0, token usage: 0.87, #running-req: 12, #queue-req: 19, #pending-token: 9529, cuda graph: True, input throughput (token/s): 802.99
[2026-10-17 15:33:07] INFO:     127.0.0.1:57920 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:07] INFO:     127.0.0.1:57936 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:07] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.81, #running-req: 12, #queue-req: 19, #pending-token: 9018, cuda graph: True, input throughput (token/s): 461.77
[2026-10-17 15:33:07] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.88, #running-req: 12, #queue-req: 19, #pending-token: 9529, cuda graph: True, input throughput (token/s): 1062.62
[2026-10-17 15:33:08] INFO:     127.0.0.1:57942 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:08] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.89, #running-req: 12, #queue-req: 19, #pending-token: 9017, cuda graph: True, input throughput (token/s): 517.21
[2026-10-17 15:33:09] INFO:     127.0.0.1:57950 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:09] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.89, #running-req: 12, #queue-req: 19, #pending-token: 9529, cuda graph: True, input throughput (token/s): 729.32
[2026-10-17 15:33:10] INFO:     127.0.0.1:57964 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:10] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.89, #running-req: 12, #queue-req: 19, #pending-token: 9529, cuda graph: True, input throughput (token/s): 554.97
[2026-10-17 15:33:10] INFO:     127.0.0.1:57980 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:10] INFO:     127.0.0.1:57990 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:11] INFO:     127.0.0.1:57992 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:11] Prefill batch, #new-seq: 2, #new-token: 1023, #cached-token: 0, token usage: 0.81, #running-req: 11, #queue-req: 19, #pending-token: 9530, cuda graph: True, input throughput (token/s): 815.80
[2026-10-17 15:33:12] Prefill batch, #new-seq: 1, #new-token: 510, #cached-token: 0, token usage: 0.87, #running-req: 12, #queue-req: 19, #pending-token: 9532, cuda graph: True, input throughput (token/s): 1025.39
[2026-10-17 15:33:12] INFO:     127.0.0.1:58000 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:12] INFO:     127.0.0.1:58016 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:13] Prefill batch, #new-seq: 2, #new-token: 1022, #cached-token: 0, token usage: 0.86, #running-req: 11, #queue-req: 19, #pending-token: 9022, cuda graph: True, input throughput (token/s): 897.79
[2026-10-17 15:33:14] Decode batch, #running-req: 13, #token: 7776, token usage: 0.88, accept len: 2.94, accept rate: 0.65, cuda graph: True, gen throughput (token/s): 175.59, #queue-req: 19
[2026-10-17 15:33:14] INFO:     127.0.0.1:58024 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:14] INFO:     127.0.0.1:58040 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:14] Prefill batch, #new-seq: 1, #new-token: 493, #cached-token: 0, token usage: 0.78, #running-req: 12, #queue-req: 19, #pending-token: 9041, cuda graph: True, input throughput (token/s): 409.02
[2026-10-17 15:33:15] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.84, #running-req: 12, #queue-req: 19, #pending-token: 9519, cuda graph: True, input throughput (token/s): 1036.29
[2026-10-17 15:33:16] INFO:     127.0.0.1:35774 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:16] INFO:     127.0.0.1:35784 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:16] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.79, #running-req: 12, #queue-req: 19, #pending-token: 9007, cuda graph: True, input throughput (token/s): 393.04
[2026-10-17 15:33:16] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.86, #running-req: 12, #queue-req: 19, #pending-token: 9519, cuda graph: True, input throughput (token/s): 1048.16
[2026-10-17 15:33:17] INFO:     127.0.0.1:35788 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:17] Prefill batch, #new-seq: 2, #new-token: 854, #cached-token: 0, token usage: 0.89, #running-req: 12, #queue-req: 18, #pending-token: 8665, cuda graph: True, input throughput (token/s): 1033.17
[2026-10-17 15:33:18] INFO:     127.0.0.1:35794 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:18] Prefill batch, #new-seq: 1, #new-token: 509, #cached-token: 0, token usage: 0.89, #running-req: 13, #queue-req: 18, #pending-token: 9179, cuda graph: True, input throughput (token/s): 627.90
[2026-10-17 15:33:19] INFO:     127.0.0.1:35800 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:19] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.91, #running-req: 13, #queue-req: 18, #pending-token: 8667, cuda graph: True, input throughput (token/s): 467.10
[2026-10-17 15:33:20] INFO:     127.0.0.1:35802 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:21] INFO:     127.0.0.1:35818 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:21] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.86, #running-req: 13, #queue-req: 18, #pending-token: 9179, cuda graph: True, input throughput (token/s): 390.52
[2026-10-17 15:33:21] INFO:     127.0.0.1:35826 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:21] INFO:     127.0.0.1:35832 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:21] INFO:     127.0.0.1:35846 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:21] Prefill batch, #new-seq: 2, #new-token: 1023, #cached-token: 0, token usage: 0.77, #running-req: 12, #queue-req: 18, #pending-token: 8668, cuda graph: True, input throughput (token/s): 1104.69
[2026-10-17 15:33:22] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.83, #running-req: 12, #queue-req: 19, #pending-token: 9691, cuda graph: True, input throughput (token/s): 1024.14
[2026-10-17 15:33:22] Decode batch, #running-req: 13, #token: 7391, token usage: 0.84, accept len: 2.92, accept rate: 0.64, cuda graph: True, gen throughput (token/s): 181.04, #queue-req: 19
[2026-10-17 15:33:22] INFO:     127.0.0.1:35854 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:22] INFO:     127.0.0.1:35860 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:24] Prefill batch, #new-seq: 3, #new-token: 1535, #cached-token: 0, token usage: 0.89, #running-req: 12, #queue-req: 18, #pending-token: 8659, cuda graph: True, input throughput (token/s): 927.59
[2026-10-17 15:33:24] INFO:     127.0.0.1:35866 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:24] INFO:     127.0.0.1:35882 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:25] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.84, #running-req: 12, #queue-req: 19, #pending-token: 9677, cuda graph: True, input throughput (token/s): 555.71
[2026-10-17 15:33:25] INFO:     127.0.0.1:47684 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:25] INFO:     127.0.0.1:47686 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:26] Prefill batch, #new-seq: 2, #new-token: 1023, #cached-token: 1, token usage: 0.83, #running-req: 11, #queue-req: 19, #pending-token: 9677, cuda graph: True, input throughput (token/s): 879.75
[2026-10-17 15:33:26] INFO:     127.0.0.1:47696 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:27] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.83, #running-req: 12, #queue-req: 19, #pending-token: 9677, cuda graph: True, input throughput (token/s): 567.68
[2026-10-17 15:33:28] INFO:     127.0.0.1:47706 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:28] Prefill batch, #new-seq: 1, #new-token: 478, #cached-token: 0, token usage: 0.87, #running-req: 12, #queue-req: 19, #pending-token: 9199, cuda graph: True, input throughput (token/s): 290.24
[2026-10-17 15:33:29] INFO:     127.0.0.1:47712 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:29] Prefill batch, #new-seq: 1, #new-token: 510, #cached-token: 2, token usage: 0.89, #running-req: 12, #queue-req: 19, #pending-token: 9664, cuda graph: True, input throughput (token/s): 426.90
[2026-10-17 15:33:30] INFO:     127.0.0.1:47716 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:30] Decode batch, #running-req: 12, #token: 7048, token usage: 0.80, accept len: 2.93, accept rate: 0.64, cuda graph: True, gen throughput (token/s): 188.78, #queue-req: 18
[2026-10-17 15:33:30] INFO:     127.0.0.1:47728 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:30] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.81, #running-req: 12, #queue-req: 19, #pending-token: 9152, cuda graph: True, input throughput (token/s): 706.07
[2026-10-17 15:33:31] INFO:     127.0.0.1:47730 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:31] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.80, #running-req: 12, #queue-req: 19, #pending-token: 9664, cuda graph: True, input throughput (token/s): 775.84
[2026-10-17 15:33:31] INFO:     127.0.0.1:47746 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:31] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.78, #running-req: 12, #queue-req: 19, #pending-token: 9665, cuda graph: True, input throughput (token/s): 776.38
[2026-10-17 15:33:32] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.85, #running-req: 12, #queue-req: 19, #pending-token: 9665, cuda graph: True, input throughput (token/s): 779.25
[2026-10-17 15:33:32] INFO:     127.0.0.1:47758 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:33] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.85, #running-req: 12, #queue-req: 19, #pending-token: 9665, cuda graph: True, input throughput (token/s): 488.97
[2026-10-17 15:33:34] INFO:     127.0.0.1:47768 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:34] INFO:     127.0.0.1:47772 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:35] Prefill batch, #new-seq: 2, #new-token: 1020, #cached-token: 4, token usage: 0.84, #running-req: 12, #queue-req: 19, #pending-token: 9153, cuda graph: True, input throughput (token/s): 617.29
[2026-10-17 15:33:35] INFO:     127.0.0.1:58956 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:36] INFO:     127.0.0.1:58962 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:36] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.77, #running-req: 12, #queue-req: 19, #pending-token: 9658, cuda graph: True, input throughput (token/s): 725.89
[2026-10-17 15:33:36] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.83, #running-req: 12, #queue-req: 19, #pending-token: 9659, cuda graph: True, input throughput (token/s): 1150.87
[2026-10-17 15:33:36] INFO:     127.0.0.1:58964 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:36] INFO:     127.0.0.1:58978 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:38] Prefill batch, #new-seq: 2, #new-token: 1015, #cached-token: 0, token usage: 0.83, #running-req: 11, #queue-req: 19, #pending-token: 9667, cuda graph: True, input throughput (token/s): 643.78
[2026-10-17 15:33:38] INFO:     127.0.0.1:58986 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:39] Prefill batch, #new-seq: 1, #new-token: 501, #cached-token: 6, token usage: 0.85, #running-req: 12, #queue-req: 19, #pending-token: 9160, cuda graph: True, input throughput (token/s): 337.58
[2026-10-17 15:33:39] INFO:     127.0.0.1:58988 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:40] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.85, #running-req: 12, #queue-req: 19, #pending-token: 9673, cuda graph: True, input throughput (token/s): 632.47
[2026-10-17 15:33:40] Decode batch, #running-req: 13, #token: 6942, token usage: 0.79, accept len: 2.73, accept rate: 0.58, cuda graph: True, gen throughput (token/s): 139.04, #queue-req: 19
[2026-10-17 15:33:40] INFO:     127.0.0.1:58990 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:41] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 1, token usage: 0.86, #running-req: 12, #queue-req: 19, #pending-token: 9673, cuda graph: True, input throughput (token/s): 508.93
[2026-10-17 15:33:42] INFO:     127.0.0.1:58998 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:42] INFO:     127.0.0.1:59002 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:42] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.80, #running-req: 12, #queue-req: 19, #pending-token: 9673, cuda graph: True, input throughput (token/s): 396.59
[2026-10-17 15:33:43] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.87, #running-req: 12, #queue-req: 19, #pending-token: 9673, cuda graph: True, input throughput (token/s): 1102.39
[2026-10-17 15:33:43] INFO:     127.0.0.1:59018 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:43] Prefill batch, #new-seq: 1, #new-token: 509, #cached-token: 3, token usage: 0.86, #running-req: 12, #queue-req: 19, #pending-token: 9161, cuda graph: True, input throughput (token/s): 905.85
[2026-10-17 15:33:44] INFO:     127.0.0.1:59026 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:44] INFO:     127.0.0.1:59028 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:44] Prefill batch, #new-seq: 1, #new-token: 499, #cached-token: 0, token usage: 0.79, #running-req: 12, #queue-req: 19, #pending-token: 9174, cuda graph: True, input throughput (token/s): 608.92
[2026-10-17 15:33:44] Prefill batch, #new-seq: 1, #new-token: 478, #cached-token: 0, token usage: 0.85, #running-req: 12, #queue-req: 19, #pending-token: 9720, cuda graph: True, input throughput (token/s): 1024.88
[2026-10-17 15:33:45] INFO:     127.0.0.1:48228 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:45] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.85, #running-req: 12, #queue-req: 19, #pending-token: 9720, cuda graph: True, input throughput (token/s): 632.43
[2026-10-17 15:33:46] INFO:     127.0.0.1:48244 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:47] INFO:     127.0.0.1:48254 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:47] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.78, #running-req: 12, #queue-req: 19, #pending-token: 9720, cuda graph: True, input throughput (token/s): 362.99
[2026-10-17 15:33:47] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.85, #running-req: 12, #queue-req: 19, #pending-token: 9720, cuda graph: True, input throughput (token/s): 724.50
[2026-10-17 15:33:48] Decode batch, #running-req: 13, #token: 7742, token usage: 0.88, accept len: 2.67, accept rate: 0.56, cuda graph: True, gen throughput (token/s): 168.25, #queue-req: 19
[2026-10-17 15:33:48] INFO:     127.0.0.1:48266 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:48] INFO:     127.0.0.1:48282 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:50] Prefill batch, #new-seq: 2, #new-token: 1022, #cached-token: 2, token usage: 0.86, #running-req: 12, #queue-req: 19, #pending-token: 9208, cuda graph: True, input throughput (token/s): 478.52
[2026-10-17 15:33:50] INFO:     127.0.0.1:48288 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:50] INFO:     127.0.0.1:48296 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:51] INFO:     127.0.0.1:48300 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:51] Prefill batch, #new-seq: 2, #new-token: 1021, #cached-token: 3, token usage: 0.77, #running-req: 11, #queue-req: 19, #pending-token: 9208, cuda graph: True, input throughput (token/s): 717.61
[2026-10-17 15:33:51] Prefill batch, #new-seq: 1, #new-token: 505, #cached-token: 0, token usage: 0.83, #running-req: 12, #queue-req: 19, #pending-token: 9724, cuda graph: True, input throughput (token/s): 1097.77
[2026-10-17 15:33:52] INFO:     127.0.0.1:48308 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:52] INFO:     127.0.0.1:48310 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:53] Prefill batch, #new-seq: 2, #new-token: 1023, #cached-token: 1, token usage: 0.85, #running-req: 12, #queue-req: 19, #pending-token: 9212, cuda graph: True, input throughput (token/s): 537.42
[2026-10-17 15:33:54] INFO:     127.0.0.1:48326 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:54] INFO:     127.0.0.1:48328 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:33:55] Prefill batch, #new-seq: 3, #new-token: 1531, #cached-token: 4, token usage: 0.82, #running-req: 12, #queue-req: 18, #pending-token: 8189, cuda graph: True, input throughput (token/s): 924.67
[2026-10-17 15:33:56] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.84, #running-req: 12, #queue-req: 17, #pending-token: 8701, cuda graph: True, input throughput (token/s): 561.41
[2026-10-17 15:33:57] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.85, #running-req: 12, #queue-req: 16, #pending-token: 8189, cuda graph: True, input throughput (token/s): 526.19
[2026-10-17 15:33:58] Decode batch, #running-req: 13, #token: 7003, token usage: 0.79, accept len: 2.77, accept rate: 0.59, cuda graph: True, gen throughput (token/s): 153.15, #queue-req: 16
[2026-10-17 15:33:58] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 1, token usage: 0.86, #running-req: 12, #queue-req: 15, #pending-token: 7677, cuda graph: True, input throughput (token/s): 382.25
[2026-10-17 15:33:59] Prefill batch, #new-seq: 2, #new-token: 1023, #cached-token: 1, token usage: 0.85, #running-req: 11, #queue-req: 13, #pending-token: 6653, cuda graph: True, input throughput (token/s): 833.19
[2026-10-17 15:34:01] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.85, #running-req: 12, #queue-req: 12, #pending-token: 6141, cuda graph: True, input throughput (token/s): 460.94
[2026-10-17 15:34:02] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.85, #running-req: 12, #queue-req: 11, #pending-token: 5629, cuda graph: True, input throughput (token/s): 510.33
[2026-10-17 15:34:03] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.79, #running-req: 12, #queue-req: 10, #pending-token: 5117, cuda graph: True, input throughput (token/s): 454.60
[2026-10-17 15:34:03] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.85, #running-req: 12, #queue-req: 9, #pending-token: 4605, cuda graph: True, input throughput (token/s): 885.92
[2026-10-17 15:34:05] Prefill batch, #new-seq: 2, #new-token: 1023, #cached-token: 1, token usage: 0.85, #running-req: 12, #queue-req: 7, #pending-token: 3581, cuda graph: True, input throughput (token/s): 575.31
[2026-10-17 15:34:06] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.87, #running-req: 12, #queue-req: 6, #pending-token: 3069, cuda graph: True, input throughput (token/s): 381.63
[2026-10-17 15:34:06] Decode batch, #running-req: 13, #token: 7658, token usage: 0.87, accept len: 2.55, accept rate: 0.52, cuda graph: True, gen throughput (token/s): 145.36, #queue-req: 6
[2026-10-17 15:34:07] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.88, #running-req: 12, #queue-req: 5, #pending-token: 2557, cuda graph: True, input throughput (token/s): 452.90
[2026-10-17 15:34:09] Prefill batch, #new-seq: 2, #new-token: 1019, #cached-token: 2, token usage: 0.87, #running-req: 12, #queue-req: 3, #pending-token: 1536, cuda graph: True, input throughput (token/s): 762.91
[2026-10-17 15:34:10] Prefill batch, #new-seq: 2, #new-token: 1022, #cached-token: 2, token usage: 0.79, #running-req: 12, #queue-req: 1, #pending-token: 512, cuda graph: True, input throughput (token/s): 654.92
[2026-10-17 15:34:11] Prefill batch, #new-seq: 1, #new-token: 508, #cached-token: 4, token usage: 0.85, #running-req: 12, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 882.17
[2026-10-17 15:34:14] Decode batch, #running-req: 6, #token: 3569, token usage: 0.40, accept len: 2.76, accept rate: 0.59, cuda graph: True, gen throughput (token/s): 171.47, #queue-req: 0
[2026-10-17 15:34:14] Triton kernel 'assign_req_to_token_pool' device-loaded after serving started (free device mem: 0.85 GiB). Pre-load it during engine init to avoid CUDA OOM.
[2026-10-17 15:34:16] INFO:     127.0.0.1:33460 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-17 15:34:17] INFO:     127.0.0.1:33462 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-17 15:34:19] INFO:     127.0.0.1:33464 - "GET /metrics HTTP/1.1" 200 OK
[2026-10-17 15:34:19] SIGTERM received. signum=None frame=None. Draining requests and shutting down...
[2026-10-17 15:34:19] Subprocess detokenizer (pid=1342111) crashed with exit code -15. Triggering SIGQUIT for cleanup...
[2026-10-17 15:34:19] SIGQUIT received. signum=None, frame=None. It usually means one child failed.
[2026-10-17 15:34:19] Sleeping 5 seconds before crash diagnostics to let GPU activity settle.
[2026-10-17 15:34:24] No live scheduler processes found; skipping py-spy and CUDA coredump.
[2026-10-17 15:34:24] kill_process_tree called: parent_pid=1341742, include_parent=True, pid=1341742