[2026-10-17 15:14:56] Registered linear attn model: config=GigaChat35Config, backend=GDNAttnBackend, archs=['GigaChat35ForCausalLM', 'GigaChat35ForCausalLMNextN']
[2026-10-17 15:14:58] Failed to get GPU memory capacity from nvidia-smi. Falling back to torch.cuda.mem_get_info(). Reported total GPU memory per device (MiB): [124610], using min: 124610 MiB.
[2026-10-17 15:14:58] Attention backend not specified. Use flashinfer backend by default.
[2026-10-17 15:14:58] Max running requests is reset to 48 for speculative decoding. You can override this by explicitly setting --max-running-requests.
[2026-10-17 15:14:59] server_args={'model_path': 'Qwen/Qwen3-30B-A3B-Instruct-2507', 'tokenizer_path': 'Qwen/Qwen3-30B-A3B-Instruct-2507', 'tokenizer_mode': 'auto', 'tokenizer_backend': 'huggingface', 'tokenizer_worker_num': 1, 'detokenizer_worker_num': 1, 'skip_tokenizer_init': False, 'load_format': 'auto', 'model_loader_extra_config': '{}', 'trust_remote_code': False, 'context_length': None, 'is_embedding': False, 'enable_multimodal': None, 'revision': None, 'model_impl': 'auto', 'model_config_parser': 'auto', 'json_model_override_args': '{}', 'dtype': 'auto', 'quantization': None, 'quantization_param_path': None, 'kv_cache_dtype': 'auto', 'enable_fp32_lm_head': False, 'modelopt_quant': None, 'modelopt_checkpoint_restore_path': None, 'modelopt_checkpoint_save_path': None, 'modelopt_export_path': None, 'quantize_and_serve': False, 'rl_quant_profile': None, 'enable_tf32_matmul': False, 'mem_fraction_static': 0.7, 'max_running_requests': 48, 'max_queued_requests': None, 'max_total_tokens': None, 'chunked_prefill_size': 8192, 'prefill_decode_interval': 0, 'enable_dynamic_chunking': False, 'max_prefill_tokens': 16384, 'prefill_max_requests': None, 'schedule_policy': 'fcfs', 'enable_priority_scheduling': False, 'disable_priority_preemption': False, 'default_priority_value': None, 'abort_on_priority_when_disabled': False, 'schedule_low_priority_values_first': False, 'priority_scheduling_preemption_threshold': 10, 'retraction_policy': 'length', 'schedule_conservativeness': 1.0, 'page_size': 1, 'c128_page_size': 16, 'swa_full_tokens_ratio': 0.8, 'disable_hybrid_swa_memory': False, 'radix_eviction_policy': 'lru', 'radix_eviction_policy_config': None, 'prefill_only_disable_kv_cache': False, 'disable_radix_cache': False, 'enable_page_major_kv_layout': False, 'enable_unified_memory': False, 'disable_chunked_prefix_cache': False, 'disable_overlap_schedule': False, 'num_continuous_decode_steps': 1, 'scheduler_recv_interval': 1, 'enable_mixed_chunk': False, 'nccl_port': None, 'dist_timeout': None, 'dist_init_addr': None, 'gated_launch_port': None, 'nnodes': 1, 'node_rank': 0, 'tp_size': 1, 'dcp_size': 1, 'pp_size': 1, 'pp_max_micro_batch_size': None, 'pp_async_batch_depth': 0, 'dp_size': 1, 'attn_dp_size': 1, 'load_balance_method': 'round_robin', 'attn_cp_size': 1, 'moe_dp_size': 1, 'dwdp_size': 1, 'dcp_comm_backend': 'ag_rs', 'dcp_replicate_q_proj': None, 'enable_prefill_cp': False, 'cp_strategy': None, 'enable_dsa_cache_layer_split': False, 'enable_cp_decode_attn_tp': False, 'enable_kv_cache_sharding': False, 'enable_dp_attention': False, 'enable_dp_attention_local_control_broadcast': False, 'enable_dp_lm_head': False, 'enable_tp_lm_head_all_to_all': False, 'enable_attn_tp_input_scattered': False, 'enable_shared_experts_attn_tp': False, 'shared_experts_tp_size': None, 'enable_dense_mlp_attn_tp': False, 'enable_layernorm_sp': False, 'disable_attn_tp_gather': False, 'enable_p2p_check': False, 'device': 'cuda', 'base_gpu_id': 0, 'gpu_id_step': 1, 'random_seed': 776105296, 'mlx_enable_sampling': False, 'watchdog_timeout': 300, 'soft_watchdog_timeout': None, 'sleep_on_idle': False, 'use_ray': False, 'custom_sigquit_handler': None, 'numa_node': None, 'gc_threshold': None, 'host': '127.0.0.1', 'port': 30000, 'fastapi_root_path': '', 'smg_grpc_mode': False, 'grpc_mode': False, 'grpc_port': None, 'grpc_worker_threads': 4, 'sidecar': None, 'sidecar_args': None, 'skip_server_warmup': False, 'warmups': None, 'enable_http2': False, 'http2_max_concurrent_streams': 200, 'http2_initial_connection_window_size': 1048576, 'ssl_keyfile': None, 'ssl_certfile': None, 'ssl_ca_certs': None, 'ssl_keyfile_password': None, 'enable_ssl_refresh': False, 'api_key': None, 'admin_api_key': None, 'served_model_name': 'Qwen/Qwen3-30B-A3B-Instruct-2507', 'weight_version': 'default', 'chat_template': None, 'trust_request_chat_template': False, 'hf_chat_template_name': None, 'completion_template': None, 'file_storage_path': None, 'enable_cache_report': True, 'return_input_ids': False, 'return_output_ids': False, 'reasoning_parser': None, 'default_chat_template_kwargs': None, 'strip_thinking_cache': False, 'enable_strict_thinking': False, 'tool_call_parser': None, 'tool_server': None, 'sampling_defaults': 'model', 'asr_max_buffer_seconds': 60, 'asr_max_concurrent_sessions': 32, 'preferred_sampling_params': None, 'allow_auto_truncate': False, 'stream_interval': 1, 'batch_notify_size': 16, 'stream_response_default_include_usage': False, 'incremental_streaming_output': False, 'enable_streaming_session': False, 'enable_session_radix_cache': False, 'log_level': 'info', 'log_level_http': None, 'log_requests': False, 'log_requests_level': 2, 'log_requests_format': 'text', 'log_requests_target': None, 'uvicorn_access_log_exclude_prefixes': [], 'crash_dump_folder': None, 'show_time_cost': False, 'enable_metrics': True, 'smg_http_sidecar_port': None, 'enable_mfu_metrics': False, 'enable_metrics_for_all_schedulers': False, 'load_snapshot_publish_interval': 15, 'tokenizer_metrics_custom_labels_header': 'x-custom-labels', 'tokenizer_metrics_allowed_custom_labels': None, 'extra_metric_labels': None, 'bucket_time_to_first_token': None, 'bucket_inter_token_latency': None, 'bucket_e2e_request_latency': None, 'prompt_tokens_buckets': None, 'generation_tokens_buckets': None, 'gc_warning_threshold_secs': 0.0, 'decode_log_interval': 40, 'enable_request_time_stats_logging': False, 'kv_events_config': None, 'load_publish_endpoint': None, 'enable_forward_pass_metrics': False, 'forward_pass_metrics_worker_id': '', 'forward_pass_metrics_ipc_name': None, 'enable_trace': False, 'trace_modules': 'request', 'otlp_traces_endpoint': 'localhost:4317', 'export_metrics_to_file': False, 'export_metrics_to_file_dir': None, 'stat_loggers': None, 'constrained_json_whitespace_pattern': None, 'constrained_json_disable_any_whitespace': False, 'constrained_json_max_whitespace_cnt': None, 'attention_backend': 'flashinfer', 'decode_attention_backend': None, 'enable_lean_attention': None, 'prefill_attention_backend': None, 'sampling_backend': 'flashinfer', 'grammar_backend': 'xgrammar', 'radix_cache_backend': None, 'mm_attention_backend': None, 'fp8_gemm_runner_backend': 'auto', 'fp4_gemm_runner_backend': 'auto', 'bf16_gemm_backend': 'auto', 'dsa_prefill_backend': None, 'dsv4_prefill_backend': 'auto', 'dsa_decode_backend': None, 'dsa_paged_mqa_logits_backend': 'auto', 'dsa_topk_backend': 'sgl-kernel', 'disable_flashinfer_autotune': False, 'flashinfer_autotune_skip_ops': None, 'mamba_backend': 'triton', 'cuda_graph_config': {'decode': {'backend': 'full', 'max_bs': 256, 'bs': [1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 40, 44, 48, 52, 56, 60, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'tc_compiler': 'eager', 'max_context_size': None, 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None, 'max_seq_len': None}, 'prefill': {'backend': 'breakable', 'max_bs': 8192, 'bs': [4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096, 4608, 5120, 5632, 6144, 6656, 7168, 7680, 8192], 'tc_compiler': 'eager', 'max_context_size': None, 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None, 'max_seq_len': None}}, 'cuda_graph_backend_decode': None, 'cuda_graph_backend_prefill': None, 'cuda_graph_max_bs_decode': None, 'cuda_graph_max_bs_prefill': None, 'cuda_graph_bs_decode': None, 'cuda_graph_bs_prefill': None, 'cuda_graph_prefill_max_context': None, 'disable_prefill_cuda_graph': False, 'disable_decode_cuda_graph': False, 'disable_cuda_graph': False, 'disable_cuda_graph_padding': False, 'enable_profile_cuda_graph': False, 'enable_cudagraph_gc': False, 'debug_cuda_graph': False, 'enable_layerwise_nvtx_marker': False, 'enable_nccl_nvls': False, 'enable_symm_mem': False, 'triton_attention_reduce_in_fp32': False, 'triton_attention_num_kv_splits': 8, 'triton_attention_split_tile_size': None, 'flashinfer_mla_disable_ragged': False, 'enable_fused_qk_norm_rope': False, 'enable_precise_embedding_interpolation': False, 'enable_fused_moe_sum_all_reduce': False, 'enable_deepseek_v4_fp4_indexer': False, 'disable_custom_all_reduce': False, 'enable_mscclpp': False, 'enable_torch_symm_mem': False, 'enable_scattered_sconv': False, 'pre_warm_nccl': False, 'boundary_reduction': 'rs+rsv', 'enable_quant_communications': False, 'enable_flashinfer_allreduce_fusion': False, 'enforce_disable_flashinfer_allreduce_fusion': False, 'flashinfer_allreduce_fusion_backend': None, 'enable_aiter_allreduce_fusion': False, 'enable_torch_compile': False, 'enable_torch_compile_debug_mode': False, 'torch_compile_max_bs': 32, 'speculative_algorithm': 'EAGLE3', 'uno_lora_path': None, 'speculative_draft_model_path': 'lmsys/SGLang-EAGLE3-Qwen3-30B-A3B-Instruct-2507-SpecForge-Nex', 'speculative_draft_model_revision': 'main', 'speculative_draft_load_format': None, 'speculative_num_steps': 3, 'speculative_eagle_topk': 1, 'speculative_num_draft_tokens': 4, 'speculative_dflash_block_size': None, 'speculative_dspark_block_size': None, 'speculative_dspark_sps_table_path': None, 'speculative_dspark_confidence_sts_path': None, 'speculative_dspark_align_verify_tokens_to_graph_tier': False, 'speculative_accept_threshold_single': 1.0, 'speculative_accept_threshold_acc': 1.0, 'speculative_use_rejection_sampling': False, 'speculative_token_map': None, 'speculative_attention_mode': 'prefill', 'speculative_draft_attention_backend': None, 'speculative_dsa_topk_backend': 'sgl-kernel', 'speculative_draft_kv_cache_dtype': None, 'speculative_draft_window_size': None, 'speculative_moe_runner_backend': 'auto', 'speculative_moe_a2a_backend': None, 'speculative_enable_w4a4_mxfp4_megamoe': None, 'speculative_draft_model_quantization': None, '_speculative_draft_quantization_explicitly_set': False, 'speculative_skip_dp_mlp_sync': False, 'enable_multi_layer_eagle': False, 'speculative_adaptive': False, 'speculative_adaptive_config': None, 'decoupled_spec_bind_endpoint': None, 'decoupled_spec_connect_endpoints': None, 'decoupled_spec_rank': None, 'decoupled_spec_role': 'null', 'spec_trace_dir': None, 'speculative_ngram_min_bfs_breadth': 1, 'speculative_ngram_max_bfs_breadth': 10, 'speculative_ngram_match_type': 'BFS', 'speculative_ngram_max_trie_depth': 18, 'speculative_ngram_capacity': 10000000, 'speculative_ngram_external_corpus_path': None, 'speculative_ngram_external_sam_budget': 0, 'speculative_ngram_external_corpus_max_tokens': 10000000, 'ep_size': 1, 'moe_a2a_backend': 'none', 'enable_w4a4_mxfp4_megamoe': False, 'deepep_v2_mode': 'direct', 'moe_runner_backend': 'auto', 'flashinfer_mxfp4_moe_precision': 'default', 'deepep_mode': 'auto', 'fuseep_mode': 2, 'deepep_dispatcher_output_dtype': 'auto', 'ep_num_redundant_experts': 0, 'ep_dispatch_algorithm': None, 'init_expert_location': 'trivial', 'enable_eplb': False, 'eplb_algorithm': 'auto', 'eplb_rebalance_num_iterations': 1000, 'eplb_rebalance_layers_per_chunk': None, 'eplb_min_rebalancing_utilization_threshold': 1.0, 'expert_distribution_recorder_mode': None, 'expert_distribution_recorder_buffer_size': 1000, 'expert_balancedness_report_mode': 'off', 'deepep_config': None, 'moe_dense_tp_size': None, 'elastic_ep_backend': None, 'enable_elastic_expert_backup': False, 'mooncake_ib_device': None, 'enable_waterfill': False, 'ep_join_mode': None, 'ep_join_rank_offset': 0, 'elastic_ep_initial_size': None, 'max_ep_size': None, 'elastic_ep_scale_timeout': 600, 'disable_flashinfer_cutlass_moe_fp4_allgather': False, 'disable_shared_experts_fusion': False, 'enforce_shared_experts_fusion': False, 'max_mamba_cache_size': None, 'mamba_ssm_dtype': None, 'mamba_max_states_per_path': -1, 'enable_mamba_cache_stochastic_rounding': False, 'mamba_cache_philox_rounds': 0, 'mamba_full_memory_ratio': 0.9, 'mamba_radix_cache_strategy': 'auto', 'uses_mamba_radix_cache': False, 'mamba_track_interval': 256, 'enable_int8_mamba_checkpoint': False, 'int8_mamba_ckpt_size': None, 'linear_attn_backend': 'triton', 'linear_attn_decode_backend': None, 'linear_attn_prefill_backend': None, 'linear_attn_verify_backend': None, 'enable_linear_replayssm': False, 'linear_replayssm_cache_len': 16, 'enable_linear_replayssm_spec': False, 'enable_hierarchical_cache': False, 'hicache_host_memory_mode': 'cache', 'hicache_ratio': 2.0, 'hicache_size': 0, 'hicache_write_policy': 'write_through', 'hicache_io_backend': 'kernel', 'hicache_mem_layout': 'page_first', 'hicache_storage_backend': None, 'hicache_storage_prefetch_policy': 'timeout', 'hicache_storage_backend_extra_config': None, 'hicache_storage_prefetch_retry_poll_interval': 8, 'hicache_storage_prefetch_retry_max_attempts': 8, 'enable_unified_cache_external_linker': False, 'unified_cache_external_linker_backend': 'mooncake', 'enable_hisparse': False, 'hisparse_config': None, 'enable_broadcast_mm_inputs_process': False, 'enable_prefix_mm_cache': False, 'mm_enable_dp_encoder': False, 'mm_process_config': {}, 'mm_processor_worker_num': 0, 'mm_io_worker_num': 0, 'allowed_media_domains': [], 'media_url_max_file_size_mb': 64, 'mm_preprocess_cache_size_mb': None, 'trust_mm_content_hashes': False, 'limit_mm_data_per_request': None, 'enable_mm_global_cache': False, 'image_processor_backend': 'auto', 'mm_global_cache_backend': 'mooncake', 'disable_fast_image_processor': False, 'mm_feature_transport': 'cpu', 'enable_lora': None, 'enable_lora_overlap_loading': None, 'max_lora_rank': None, 'lora_target_modules': None, 'lora_paths': None, 'max_loaded_loras': None, 'max_loras_per_batch': 8, 'lora_eviction_policy': 'lru', 'lora_backend': 'csgmv', 'max_lora_chunk_size': 16, 'experts_shared_outer_loras': None, 'lora_use_virtual_experts': False, 'lora_strict_loading': False, 'lora_drain_wait_threshold': 0.0, 'enable_two_batch_overlap': False, 'enable_single_batch_overlap': False, 'tbo_token_distribution_threshold': 0.48, 'cpu_offload_gb': 0, 'offload_group_size': -1, 'offload_num_in_group': 1, 'offload_prefetch_step': 1, 'offload_mode': 'cpu', 'enable_lmcache': False, 'lmcache_config_file': None, 'enable_flexkv': False, 'flexkv_config_file': None, 'kt_weight_path': None, 'kt_method': 'AMXINT4', 'kt_cpuinfer': None, 'kt_threadpool_count': 2, 'kt_num_gpu_experts': None, 'kt_max_deferred_experts_per_token': None, 'dllm_algorithm': None, 'dllm_algorithm_config': None, 'dllm_fdfo': True, 'disaggregation_mode': 'null', 'disaggregation_transfer_backend': 'mooncake', 'disaggregation_bootstrap_port': 8998, 'disaggregation_ib_device': None, 'disaggregation_decode_enable_radix_cache': False, 'disaggregation_decode_enable_offload_kvcache': False, 'disaggregation_decode_retraction_backup': None, 'num_reserved_decode_tokens': 512, 'disaggregation_decode_extra_slots': None, 'disaggregation_decode_polling_interval': 1, 'optimistic_prefill_attempts': 0, 'encoder_only': False, 'language_only': False, 'language_model_only': False, 'encoder_transfer_backend': 'zmq_to_scheduler', 'encoder_urls': [], 'encoder_bootstrap_port': 8997, 'encoder_register_urls': [], 'enable_adaptive_dispatch_to_encoder': False, 'enable_pdmux': False, 'pdmux_config_path': None, 'sm_group_num': 8, 'startup_weight_load_mode': 'serial', 'custom_weight_loader': [], 'weight_loader_disable_mmap': False, 'weight_loader_prefetch_checkpoints': False, 'weight_loader_prefetch_num_threads': 4, 'weight_loader_drop_cache_after_load': False, 'remote_instance_weight_loader_seed_instance_ip': None, 'remote_instance_weight_loader_seed_instance_service_port': None, 'remote_instance_weight_loader_send_weights_group_ports': None, 'remote_instance_weight_loader_backend': 'nccl', 'remote_instance_weight_loader_start_seed_via_transfer_engine': False, 'engine_info_bootstrap_port': 6789, 'modelexpress_config': None, 'download_dir': None, 'model_checksum': None, 'delete_ckpt_after_loading': False, 'decrypted_config_file': None, 'decrypted_draft_config_file': None, 'checkpoint_engine_wait_weights_before_ready': False, 'enable_prefill_delayer': False, 'prefill_delayer_max_delay_passes': 30, 'prefill_delayer_token_usage_low_watermark': None, 'prefill_delayer_forward_passes_buckets': None, 'prefill_delayer_wait_seconds_buckets': None, 'prefill_delayer_queue_min_ratio': None, 'prefill_delayer_max_delay_ms': None, 'min_free_slots_delay': None, 'enable_deterministic_inference': False, 'rl_on_policy_target': None, 'kv_canary': 'none', 'kv_canary_real_data': 'none', 'kv_canary_sweep_interval': 0, 'enable_dynamic_batch_tokenizer': False, 'dynamic_batch_tokenizer_batch_size': 32, 'dynamic_batch_tokenizer_batch_timeout': 0.002, 'enable_tokenizer_batch_encode': False, 'disable_tokenizer_batch_decode': False, 'debug_tensor_dump_output_folder': None, 'debug_tensor_dump_layers': None, 'debug_tensor_dump_input_file': None, 'enable_memory_saver': False, 'enable_weights_cpu_backup': False, 'enable_draft_weights_cpu_backup': False, 'enable_custom_logit_processor': False, 'enable_return_hidden_states': False, 'return_hidden_states_mode': None, 'enable_return_routed_experts': False, 'enable_return_indexer_topk': False, 'sampling_mask_max_tokens': 4096, 'disable_outlines_disk_cache': False, 'enable_mis': False, 'weight_cache_mode': 'off', 'weight_cache_socket': None, 'weight_cache_timeout': 1800, 'forward_hooks': None, 'msprobe_dump_config': None, 'ple_offload_embedding': None, 'ple_offload_backend': 'pinned', 'ple_offload_dir': None, 'disaggregation_enable_kv_checksum': False, 'enable_pd_role_switch': False, 'speculative_domino_candidate_pool_size': 2048, 'flashinfer_a2a_dispatch_type': None, 'cuda_graph_max_seq_len_prefill': None, 'prefill_kv_cache_dequant_dtype': 'auto', 'dsv4_attn_backend': 'auto', 'otlp_service_name': None, '_radix_eviction_policy_explicitly_set': False, 'hicache_host_memory_fraction': 0.8, 'enable_linker_mla_dedup': False, '_swa_full_tokens_ratio_explicitly_set': False, 'swa_prefix_tails': None, 'enable_encoder_swa_bounded_replay': False, 'enable_decoder_swa_bounded_replay': False, 'enable_response_store': False, 'disaggregation_decode_host_receive_threshold': 1.0, 'qsa_indexer_dtype': 'auto', 'disaggregation_decode_allocation_policy': 'early', 'speculative_boundary_reduction': 'rs+rsv', 'speculative_use_block_verification': False, 'speculative_draft_sink_size': None, 'disable_aiter_allreduce_fusion_in_prefill': False, 'disable_aiter_allreduce_fusion_in_decode': False, 'dsa_triton_union': 0, 'enable_dsa_fused_indexer': None, 'enable_cp_tp_group_sharing': False, 'enable_watermark': False, 'watermark_config': '<redacted>', 'watermark_key': '<redacted>', 'watermark_key_b': '<redacted>', 'watermark_context_window': 4, 'watermark_mixing_probability': 0.5, 'watermark_max_probability': 1.0, 'watermark_default_enabled': False, 'watermark_enforce_all': False, 'grpc_response_timeout_secs': 300}
[2026-10-17 15:15:00] Using default HuggingFace chat template with detected content format: string
[2026-10-17 15:15:04] Init parallel begin.
[2026-10-17 15:15:04] Init parallel ends. elapsed=0.26 s
[2026-10-17 15:15:05] Memory baseline taken. mem usage=0.00 GB
/home/rohit/sglang-lab/.venv/lib/python3.12/site-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`.
  warnings.warn(
/home/rohit/sglang-lab/.venv/lib/python3.12/site-packages/torch/jit/_script.py:1643: FutureWarning: `torch.jit.interface` is deprecated. Please use `torch.compile` instead.
  warnings.warn(
/home/rohit/sglang-lab/.venv/lib/python3.12/site-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`.
  warnings.warn(
[2026-10-17 15:15:06] Load weight begin. avail mem=113.45 GB
[2026-10-17 15:15:08] Found local HF snapshot for Qwen/Qwen3-30B-A3B-Instruct-2507 at /home/rohit/sglang-lab/hf/hub/models--Qwen--Qwen3-30B-A3B-Instruct-2507/snapshots/0d7cf23991f47feeb3a57ecb4c9cee8ea4a17bfe; skipping download.

Multi-thread loading shards:   0% Completed | 0/16 [00:00<?, ?it/s]
Multi-thread loading shards:   6% Completed | 1/16 [00:21<05:18, 21.25s/it]
Multi-thread loading shards:  12% Completed | 2/16 [00:41<04:49, 20.67s/it]
Multi-thread loading shards:  19% Completed | 3/16 [01:01<04:24, 20.35s/it]
Multi-thread loading shards:  25% Completed | 4/16 [01:21<04:04, 20.34s/it]
Multi-thread loading shards:  31% Completed | 5/16 [01:42<03:43, 20.33s/it]
Multi-thread loading shards:  38% Completed | 6/16 [02:02<03:24, 20.41s/it]
Multi-thread loading shards:  44% Completed | 7/16 [02:22<03:03, 20.37s/it]
Multi-thread loading shards:  50% Completed | 8/16 [02:43<02:42, 20.28s/it]
Multi-thread loading shards:  56% Completed | 9/16 [03:02<02:21, 20.17s/it]
Multi-thread loading shards:  62% Completed | 10/16 [03:23<02:00, 20.14s/it]
Multi-thread loading shards:  69% Completed | 11/16 [03:43<01:41, 20.30s/it]
Multi-thread loading shards:  75% Completed | 12/16 [04:03<01:21, 20.28s/it]
Multi-thread loading shards:  81% Completed | 13/16 [04:24<01:00, 20.27s/it]
Multi-thread loading shards:  88% Completed | 14/16 [04:44<00:40, 20.24s/it]
Multi-thread loading shards:  94% Completed | 15/16 [05:04<00:20, 20.08s/it]
Multi-thread loading shards: 100% Completed | 16/16 [05:10<00:00, 15.83s/it]
Multi-thread loading shards: 100% Completed | 16/16 [05:10<00:00, 19.38s/it]
[2026-10-17 15:20:18] Load weight end. elapsed=312.38 s, type=Qwen3MoeForCausalLM, avail mem=55.96 GB, mem usage=57.49 GB.
[2026-10-17 15:20:19] Warning: Target model's context_length (262144) is greater than the derived context_length (2048). This may lead to incorrect model outputs or CUDA errors. Note that the derived context_length may differ from max_position_embeddings in the model's config.
[2026-10-17 15:20:19] Overriding the draft model's max_position_embeddings to 262144.
[2026-10-17 15:20:19] Memory baseline taken. mem usage=0.00 GB
[2026-10-17 15:20:19] Load weight begin. avail mem=55.74 GB
[2026-10-17 15:20:19] Found local HF snapshot for lmsys/SGLang-EAGLE3-Qwen3-30B-A3B-Instruct-2507-SpecForge-Nex at /home/rohit/sglang-lab/hf/hub/models--lmsys--SGLang-EAGLE3-Qwen3-30B-A3B-Instruct-2507-SpecForge-Nex/snapshots/d1ac703a537d2b8a5b748d4f5f8ca7e97efe9214; skipping download.

Multi-thread loading shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Multi-thread loading shards: 100% Completed | 1/1 [00:01<00:00,  1.18s/it]
Multi-thread loading shards: 100% Completed | 1/1 [00:01<00:00,  1.18s/it]
[2026-10-17 15:20:21] Load weight end. elapsed=1.30 s, type=LlamaForCausalLMEagle3, avail mem=54.16 GB, mem usage=1.58 GB.
[2026-10-17 15:20:23] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 216775, K size: 9.92 GB, V size: 9.92 GB
[2026-10-17 15:20:23] Memory pool end. avail mem=34.44 GB
[2026-10-17 15:20:23] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 216775, K size: 0.21 GB, V size: 0.21 GB
[2026-10-17 15:20:23] Memory pool end. avail mem=33.59 GB
[2026-10-17 15:20:24] Capture target prefill CUDA graph begin. backend=breakable, num_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096, 4608, 5120, 5632, 6144, 6656, 7168, 7680, 8192], avail mem=33.84 GB

  0%|          | 0/58 [00:00<?, ?it/s]
Capturing prefill shape (num_tokens=8192 avail_mem=33.83 GB):   0%|          | 0/58 [00:00<?, ?it/s][2026-10-17 15:20:26] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/rohit/sglang-lab/.venv/lib/python3.12/site-packages/sglang/srt/layers/moe/moe_runner/triton_utils/configs/triton_3_8_0/E=128,N=768,device_name=NVIDIA_GB10.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
[2026-10-17 15:20:26] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/rohit/sglang-lab/.venv/lib/python3.12/site-packages/sglang/srt/layers/moe/moe_runner/triton_utils/configs/triton_3_8_0/E=128,N=768,device_name=NVIDIA_GB10_down.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton

Capturing prefill shape (num_tokens=8192 avail_mem=33.83 GB):   2%|▏         | 1/58 [00:02<02:50,  2.99s/it]
Capturing prefill shape (num_tokens=7680 avail_mem=30.61 GB):   2%|▏         | 1/58 [00:02<02:50,  2.99s/it]
Capturing prefill shape (num_tokens=7680 avail_mem=30.61 GB):   3%|▎         | 2/58 [00:05<02:40,  2.86s/it]
Capturing prefill shape (num_tokens=7168 avail_mem=30.57 GB):   3%|▎         | 2/58 [00:05<02:40,  2.86s/it]
Capturing prefill shape (num_tokens=7168 avail_mem=30.57 GB):   5%|▌         | 3/58 [00:08<02:31,  2.76s/it]
Capturing prefill shape (num_tokens=6656 avail_mem=30.55 GB):   5%|▌         | 3/58 [00:08<02:31,  2.76s/it]
Capturing prefill shape (num_tokens=6656 avail_mem=30.55 GB):   7%|▋         | 4/58 [00:10<02:21,  2.62s/it]
Capturing prefill shape (num_tokens=6144 avail_mem=30.54 GB):   7%|▋         | 4/58 [00:10<02:21,  2.62s/it]
Capturing prefill shape (num_tokens=6144 avail_mem=30.54 GB):   9%|▊         | 5/58 [00:12<02:10,  2.46s/it]
Capturing prefill shape (num_tokens=5632 avail_mem=30.52 GB):   9%|▊         | 5/58 [00:12<02:10,  2.46s/it]
Capturing prefill shape (num_tokens=5632 avail_mem=30.52 GB):  10%|█         | 6/58 [00:15<02:00,  2.31s/it]
Capturing prefill shape (num_tokens=5120 avail_mem=30.51 GB):  10%|█         | 6/58 [00:15<02:00,  2.31s/it]
Capturing prefill shape (num_tokens=5120 avail_mem=30.51 GB):  12%|█▏        | 7/58 [00:16<01:48,  2.14s/it]
Capturing prefill shape (num_tokens=4608 avail_mem=30.49 GB):  12%|█▏        | 7/58 [00:16<01:48,  2.14s/it]
Capturing prefill shape (num_tokens=4608 avail_mem=30.49 GB):  14%|█▍        | 8/58 [00:18<01:38,  1.96s/it]
Capturing prefill shape (num_tokens=4096 avail_mem=30.48 GB):  14%|█▍        | 8/58 [00:18<01:38,  1.96s/it]
Capturing prefill shape (num_tokens=4096 avail_mem=30.48 GB):  16%|█▌        | 9/58 [00:19<01:27,  1.78s/it]
Capturing prefill shape (num_tokens=3840 avail_mem=30.47 GB):  16%|█▌        | 9/58 [00:19<01:27,  1.78s/it]
Capturing prefill shape (num_tokens=3840 avail_mem=30.47 GB):  17%|█▋        | 10/58 [00:21<01:17,  1.62s/it]
Capturing prefill shape (num_tokens=3584 avail_mem=30.46 GB):  17%|█▋        | 10/58 [00:21<01:17,  1.62s/it]
Capturing prefill shape (num_tokens=3584 avail_mem=30.46 GB):  19%|█▉        | 11/58 [00:22<01:09,  1.48s/it]
Capturing prefill shape (num_tokens=3328 avail_mem=30.45 GB):  19%|█▉        | 11/58 [00:22<01:09,  1.48s/it]
Capturing prefill shape (num_tokens=3328 avail_mem=30.45 GB):  21%|██        | 12/58 [00:23<01:01,  1.34s/it]
Capturing prefill shape (num_tokens=3072 avail_mem=30.43 GB):  21%|██        | 12/58 [00:23<01:01,  1.34s/it]
Capturing prefill shape (num_tokens=3072 avail_mem=30.43 GB):  22%|██▏       | 13/58 [00:24<00:55,  1.23s/it]
Capturing prefill shape (num_tokens=2816 avail_mem=30.42 GB):  22%|██▏       | 13/58 [00:24<00:55,  1.23s/it]
Capturing prefill shape (num_tokens=2816 avail_mem=30.42 GB):  24%|██▍       | 14/58 [00:25<00:49,  1.13s/it]
Capturing prefill shape (num_tokens=2560 avail_mem=30.41 GB):  24%|██▍       | 14/58 [00:25<00:49,  1.13s/it]
Capturing prefill shape (num_tokens=2560 avail_mem=30.41 GB):  26%|██▌       | 15/58 [00:25<00:44,  1.03s/it]
Capturing prefill shape (num_tokens=2304 avail_mem=30.39 GB):  26%|██▌       | 15/58 [00:25<00:44,  1.03s/it]
Capturing prefill shape (num_tokens=2304 avail_mem=30.39 GB):  28%|██▊       | 16/58 [00:26<00:39,  1.07it/s]
Capturing prefill shape (num_tokens=2048 avail_mem=30.42 GB):  28%|██▊       | 16/58 [00:26<00:39,  1.07it/s]
Capturing prefill shape (num_tokens=2048 avail_mem=30.42 GB):  29%|██▉       | 17/58 [00:27<00:34,  1.20it/s]
Capturing prefill shape (num_tokens=1792 avail_mem=30.78 GB):  29%|██▉       | 17/58 [00:27<00:34,  1.20it/s]
Capturing prefill shape (num_tokens=1792 avail_mem=30.78 GB):  31%|███       | 18/58 [00:27<00:29,  1.34it/s]
Capturing prefill shape (num_tokens=1536 avail_mem=30.81 GB):  31%|███       | 18/58 [00:27<00:29,  1.34it/s]
Capturing prefill shape (num_tokens=1536 avail_mem=30.81 GB):  33%|███▎      | 19/58 [00:28<00:25,  1.53it/s]
Capturing prefill shape (num_tokens=1280 avail_mem=30.80 GB):  33%|███▎      | 19/58 [00:28<00:25,  1.53it/s]
Capturing prefill shape (num_tokens=1280 avail_mem=30.80 GB):  34%|███▍      | 20/58 [00:28<00:21,  1.74it/s]
Capturing prefill shape (num_tokens=1024 avail_mem=30.78 GB):  34%|███▍      | 20/58 [00:28<00:21,  1.74it/s]
Capturing prefill shape (num_tokens=1024 avail_mem=30.78 GB):  36%|███▌      | 21/58 [00:28<00:18,  2.03it/s]
Capturing prefill shape (num_tokens=960 avail_mem=30.76 GB):  36%|███▌      | 21/58 [00:28<00:18,  2.03it/s] 
Capturing prefill shape (num_tokens=960 avail_mem=30.76 GB):  38%|███▊      | 22/58 [00:29<00:15,  2.38it/s]
Capturing prefill shape (num_tokens=896 avail_mem=30.75 GB):  38%|███▊      | 22/58 [00:29<00:15,  2.38it/s]
Capturing prefill shape (num_tokens=896 avail_mem=30.75 GB):  40%|███▉      | 23/58 [00:29<00:12,  2.75it/s]
Capturing prefill shape (num_tokens=832 avail_mem=30.74 GB):  40%|███▉      | 23/58 [00:29<00:12,  2.75it/s]
Capturing prefill shape (num_tokens=832 avail_mem=30.74 GB):  41%|████▏     | 24/58 [00:29<00:10,  3.14it/s]
Capturing prefill shape (num_tokens=768 avail_mem=30.72 GB):  41%|████▏     | 24/58 [00:29<00:10,  3.14it/s]
Capturing prefill shape (num_tokens=768 avail_mem=30.72 GB):  43%|████▎     | 25/58 [00:29<00:09,  3.51it/s]
Capturing prefill shape (num_tokens=704 avail_mem=30.71 GB):  43%|████▎     | 25/58 [00:29<00:09,  3.51it/s]
Capturing prefill shape (num_tokens=704 avail_mem=30.71 GB):  45%|████▍     | 26/58 [00:29<00:08,  3.87it/s]
Capturing prefill shape (num_tokens=640 avail_mem=30.69 GB):  45%|████▍     | 26/58 [00:29<00:08,  3.87it/s]
Capturing prefill shape (num_tokens=640 avail_mem=30.69 GB):  47%|████▋     | 27/58 [00:30<00:07,  4.42it/s]
Capturing prefill shape (num_tokens=576 avail_mem=30.68 GB):  47%|████▋     | 27/58 [00:30<00:07,  4.42it/s]
Capturing prefill shape (num_tokens=576 avail_mem=30.68 GB):  48%|████▊     | 28/58 [00:30<00:06,  4.97it/s]
Capturing prefill shape (num_tokens=512 avail_mem=30.67 GB):  48%|████▊     | 28/58 [00:30<00:06,  4.97it/s]
Capturing prefill shape (num_tokens=512 avail_mem=30.67 GB):  50%|█████     | 29/58 [00:30<00:05,  5.39it/s]
Capturing prefill shape (num_tokens=480 avail_mem=30.65 GB):  50%|█████     | 29/58 [00:30<00:05,  5.39it/s]
Capturing prefill shape (num_tokens=480 avail_mem=30.65 GB):  52%|█████▏    | 30/58 [00:30<00:04,  5.81it/s]
Capturing prefill shape (num_tokens=448 avail_mem=30.64 GB):  52%|█████▏    | 30/58 [00:30<00:04,  5.81it/s]
Capturing prefill shape (num_tokens=448 avail_mem=30.64 GB):  53%|█████▎    | 31/58 [00:30<00:04,  6.26it/s]
Capturing prefill shape (num_tokens=416 avail_mem=30.63 GB):  53%|█████▎    | 31/58 [00:30<00:04,  6.26it/s]
Capturing prefill shape (num_tokens=416 avail_mem=30.63 GB):  55%|█████▌    | 32/58 [00:30<00:03,  6.75it/s]
Capturing prefill shape (num_tokens=384 avail_mem=30.61 GB):  55%|█████▌    | 32/58 [00:30<00:03,  6.75it/s]
Capturing prefill shape (num_tokens=384 avail_mem=30.61 GB):  57%|█████▋    | 33/58 [00:30<00:03,  6.82it/s]
Capturing prefill shape (num_tokens=352 avail_mem=30.60 GB):  57%|█████▋    | 33/58 [00:30<00:03,  6.82it/s]
Capturing prefill shape (num_tokens=352 avail_mem=30.60 GB):  59%|█████▊    | 34/58 [00:31<00:03,  6.94it/s]
Capturing prefill shape (num_tokens=320 avail_mem=30.59 GB):  59%|█████▊    | 34/58 [00:31<00:03,  6.94it/s]
Capturing prefill shape (num_tokens=320 avail_mem=30.59 GB):  60%|██████    | 35/58 [00:31<00:03,  6.97it/s]
Capturing prefill shape (num_tokens=288 avail_mem=30.55 GB):  60%|██████    | 35/58 [00:31<00:03,  6.97it/s]
Capturing prefill shape (num_tokens=288 avail_mem=30.55 GB):  62%|██████▏   | 36/58 [00:31<00:03,  7.24it/s]
Capturing prefill shape (num_tokens=256 avail_mem=30.55 GB):  62%|██████▏   | 36/58 [00:31<00:03,  7.24it/s]
Capturing prefill shape (num_tokens=256 avail_mem=30.55 GB):  64%|██████▍   | 37/58 [00:31<00:02,  7.30it/s]
Capturing prefill shape (num_tokens=240 avail_mem=30.25 GB):  64%|██████▍   | 37/58 [00:31<00:02,  7.30it/s]
Capturing prefill shape (num_tokens=240 avail_mem=30.25 GB):  66%|██████▌   | 38/58 [00:31<00:02,  7.33it/s]
Capturing prefill shape (num_tokens=224 avail_mem=30.50 GB):  66%|██████▌   | 38/58 [00:31<00:02,  7.33it/s]
Capturing prefill shape (num_tokens=224 avail_mem=30.50 GB):  67%|██████▋   | 39/58 [00:31<00:02,  7.45it/s]
Capturing prefill shape (num_tokens=208 avail_mem=30.50 GB):  67%|██████▋   | 39/58 [00:31<00:02,  7.45it/s]
Capturing prefill shape (num_tokens=208 avail_mem=30.50 GB):  69%|██████▉   | 40/58 [00:31<00:02,  7.64it/s]
Capturing prefill shape (num_tokens=192 avail_mem=30.48 GB):  69%|██████▉   | 40/58 [00:31<00:02,  7.64it/s]
Capturing prefill shape (num_tokens=192 avail_mem=30.48 GB):  71%|███████   | 41/58 [00:31<00:02,  7.57it/s]
Capturing prefill shape (num_tokens=176 avail_mem=30.46 GB):  71%|███████   | 41/58 [00:31<00:02,  7.57it/s]
Capturing prefill shape (num_tokens=176 avail_mem=30.46 GB):  72%|███████▏  | 42/58 [00:32<00:02,  7.88it/s]
Capturing prefill shape (num_tokens=160 avail_mem=30.45 GB):  72%|███████▏  | 42/58 [00:32<00:02,  7.88it/s]
Capturing prefill shape (num_tokens=160 avail_mem=30.45 GB):  74%|███████▍  | 43/58 [00:32<00:01,  8.10it/s]
Capturing prefill shape (num_tokens=144 avail_mem=30.43 GB):  74%|███████▍  | 43/58 [00:32<00:01,  8.10it/s]
Capturing prefill shape (num_tokens=144 avail_mem=30.43 GB):  76%|███████▌  | 44/58 [00:32<00:01,  8.09it/s]
Capturing prefill shape (num_tokens=128 avail_mem=30.42 GB):  76%|███████▌  | 44/58 [00:32<00:01,  8.09it/s]
Capturing prefill shape (num_tokens=128 avail_mem=30.42 GB):  78%|███████▊  | 45/58 [00:32<00:01,  8.04it/s]
Capturing prefill shape (num_tokens=112 avail_mem=30.40 GB):  78%|███████▊  | 45/58 [00:32<00:01,  8.04it/s]
Capturing prefill shape (num_tokens=112 avail_mem=30.40 GB):  79%|███████▉  | 46/58 [00:32<00:01,  8.39it/s]
Capturing prefill shape (num_tokens=96 avail_mem=30.39 GB):  79%|███████▉  | 46/58 [00:32<00:01,  8.39it/s] 
Capturing prefill shape (num_tokens=96 avail_mem=30.39 GB):  81%|████████  | 47/58 [00:32<00:01,  8.33it/s]
Capturing prefill shape (num_tokens=80 avail_mem=30.37 GB):  81%|████████  | 47/58 [00:32<00:01,  8.33it/s]
Capturing prefill shape (num_tokens=80 avail_mem=30.37 GB):  83%|████████▎ | 48/58 [00:32<00:01,  8.60it/s]
Capturing prefill shape (num_tokens=64 avail_mem=30.36 GB):  83%|████████▎ | 48/58 [00:32<00:01,  8.60it/s]
Capturing prefill shape (num_tokens=64 avail_mem=30.36 GB):  84%|████████▍ | 49/58 [00:32<00:01,  8.26it/s]
Capturing prefill shape (num_tokens=48 avail_mem=30.35 GB):  84%|████████▍ | 49/58 [00:32<00:01,  8.26it/s]
Capturing prefill shape (num_tokens=48 avail_mem=30.35 GB):  86%|████████▌ | 50/58 [00:33<00:00,  8.46it/s]
Capturing prefill shape (num_tokens=32 avail_mem=30.33 GB):  86%|████████▌ | 50/58 [00:33<00:00,  8.46it/s]
Capturing prefill shape (num_tokens=32 avail_mem=30.33 GB):  88%|████████▊ | 51/58 [00:33<00:00,  8.59it/s]
Capturing prefill shape (num_tokens=28 avail_mem=30.31 GB):  88%|████████▊ | 51/58 [00:33<00:00,  8.59it/s]
Capturing prefill shape (num_tokens=28 avail_mem=30.31 GB):  90%|████████▉ | 52/58 [00:33<00:00,  8.62it/s]
Capturing prefill shape (num_tokens=24 avail_mem=30.30 GB):  90%|████████▉ | 52/58 [00:33<00:00,  8.62it/s]
Capturing prefill shape (num_tokens=20 avail_mem=30.29 GB):  90%|████████▉ | 52/58 [00:33<00:00,  8.62it/s]
Capturing prefill shape (num_tokens=20 avail_mem=30.29 GB):  93%|█████████▎| 54/58 [00:33<00:00,  9.11it/s]
Capturing prefill shape (num_tokens=16 avail_mem=30.27 GB):  93%|█████████▎| 54/58 [00:33<00:00,  9.11it/s]
Capturing prefill shape (num_tokens=16 avail_mem=30.27 GB):  95%|█████████▍| 55/58 [00:33<00:00,  8.07it/s]
Capturing prefill shape (num_tokens=12 avail_mem=30.17 GB):  95%|█████████▍| 55/58 [00:33<00:00,  8.07it/s]
Capturing prefill shape (num_tokens=8 avail_mem=30.14 GB):  95%|█████████▍| 55/58 [00:33<00:00,  8.07it/s] 
Capturing prefill shape (num_tokens=8 avail_mem=30.14 GB):  98%|█████████▊| 57/58 [00:33<00:00,  8.71it/s]
Capturing prefill shape (num_tokens=4 avail_mem=30.11 GB):  98%|█████████▊| 57/58 [00:33<00:00,  8.71it/s]
Capturing prefill shape (num_tokens=4 avail_mem=30.11 GB): 100%|██████████| 58/58 [00:33<00:00,  8.65it/s]
Capturing prefill shape (num_tokens=4 avail_mem=30.11 GB): 100%|██████████| 58/58 [00:33<00:00,  1.71it/s]
[2026-10-17 15:20:59] Capture target prefill CUDA graph end. elapsed=34.46 s, mem usage=2.68 GB, avail mem=31.16 GB.
[2026-10-17 15:20:59] Capture target verify CUDA graph begin. backend=full, num_tokens_per_req=4, avail mem=31.16 GB

  0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=48 avail_mem=30.93 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=48 avail_mem=30.93 GB):   4%|▍         | 1/23 [00:00<00:03,  5.62it/s]
Capturing batches (bs=44 avail_mem=30.92 GB):   4%|▍         | 1/23 [00:00<00:03,  5.62it/s]
Capturing batches (bs=44 avail_mem=30.92 GB):   9%|▊         | 2/23 [00:00<00:03,  6.81it/s]
Capturing batches (bs=40 avail_mem=30.89 GB):   9%|▊         | 2/23 [00:00<00:03,  6.81it/s]
Capturing batches (bs=40 avail_mem=30.89 GB):  13%|█▎        | 3/23 [00:00<00:02,  7.51it/s]
Capturing batches (bs=32 avail_mem=30.88 GB):  13%|█▎        | 3/23 [00:00<00:02,  7.51it/s]
Capturing batches (bs=32 avail_mem=30.88 GB):  17%|█▋        | 4/23 [00:00<00:02,  7.90it/s]
Capturing batches (bs=30 avail_mem=30.86 GB):  17%|█▋        | 4/23 [00:00<00:02,  7.90it/s]
Capturing batches (bs=30 avail_mem=30.86 GB):  22%|██▏       | 5/23 [00:00<00:02,  8.14it/s]
Capturing batches (bs=28 avail_mem=30.84 GB):  22%|██▏       | 5/23 [00:00<00:02,  8.14it/s]
Capturing batches (bs=28 avail_mem=30.84 GB):  26%|██▌       | 6/23 [00:00<00:02,  8.40it/s]
Capturing batches (bs=26 avail_mem=30.82 GB):  26%|██▌       | 6/23 [00:00<00:02,  8.40it/s]
Capturing batches (bs=26 avail_mem=30.82 GB):  30%|███       | 7/23 [00:00<00:01,  8.54it/s]
Capturing batches (bs=24 avail_mem=30.81 GB):  30%|███       | 7/23 [00:00<00:01,  8.54it/s]
Capturing batches (bs=24 avail_mem=30.81 GB):  35%|███▍      | 8/23 [00:00<00:01,  8.73it/s]
Capturing batches (bs=22 avail_mem=30.79 GB):  35%|███▍      | 8/23 [00:00<00:01,  8.73it/s]
Capturing batches (bs=22 avail_mem=30.79 GB):  39%|███▉      | 9/23 [00:01<00:01,  8.87it/s]
Capturing batches (bs=20 avail_mem=30.75 GB):  39%|███▉      | 9/23 [00:01<00:01,  8.87it/s]
Capturing batches (bs=20 avail_mem=30.75 GB):  43%|████▎     | 10/23 [00:01<00:01,  9.04it/s]
Capturing batches (bs=18 avail_mem=30.73 GB):  43%|████▎     | 10/23 [00:01<00:01,  9.04it/s]
Capturing batches (bs=18 avail_mem=30.73 GB):  48%|████▊     | 11/23 [00:01<00:01,  9.19it/s]
Capturing batches (bs=16 avail_mem=30.69 GB):  48%|████▊     | 11/23 [00:01<00:01,  9.19it/s]
Capturing batches (bs=14 avail_mem=30.67 GB):  48%|████▊     | 11/23 [00:01<00:01,  9.19it/s]
Capturing batches (bs=14 avail_mem=30.67 GB):  57%|█████▋    | 13/23 [00:01<00:01,  9.52it/s]
Capturing batches (bs=12 avail_mem=30.63 GB):  57%|█████▋    | 13/23 [00:01<00:01,  9.52it/s]
Capturing batches (bs=10 avail_mem=30.62 GB):  57%|█████▋    | 13/23 [00:01<00:01,  9.52it/s]
Capturing batches (bs=10 avail_mem=30.62 GB):  65%|██████▌   | 15/23 [00:01<00:00,  9.78it/s]
Capturing batches (bs=8 avail_mem=30.58 GB):  65%|██████▌   | 15/23 [00:01<00:00,  9.78it/s] 
Capturing batches (bs=7 avail_mem=30.56 GB):  65%|██████▌   | 15/23 [00:01<00:00,  9.78it/s]
Capturing batches (bs=7 avail_mem=30.56 GB):  74%|███████▍  | 17/23 [00:01<00:00, 10.02it/s]
Capturing batches (bs=6 avail_mem=30.52 GB):  74%|███████▍  | 17/23 [00:01<00:00, 10.02it/s]
Capturing batches (bs=5 avail_mem=30.50 GB):  74%|███████▍  | 17/23 [00:01<00:00, 10.02it/s]
Capturing batches (bs=5 avail_mem=30.50 GB):  83%|████████▎ | 19/23 [00:02<00:00, 10.17it/s]
Capturing batches (bs=4 avail_mem=30.46 GB):  83%|████████▎ | 19/23 [00:02<00:00, 10.17it/s]
Capturing batches (bs=3 avail_mem=30.44 GB):  83%|████████▎ | 19/23 [00:02<00:00, 10.17it/s]
Capturing batches (bs=3 avail_mem=30.44 GB):  91%|█████████▏| 21/23 [00:02<00:00, 10.37it/s]
Capturing batches (bs=2 avail_mem=30.41 GB):  91%|█████████▏| 21/23 [00:02<00:00, 10.37it/s]
Capturing batches (bs=1 avail_mem=30.39 GB):  91%|█████████▏| 21/23 [00:02<00:00, 10.37it/s]
Capturing batches (bs=1 avail_mem=30.39 GB): 100%|██████████| 23/23 [00:02<00:00, 10.63it/s]
Capturing batches (bs=1 avail_mem=30.39 GB): 100%|██████████| 23/23 [00:02<00:00,  9.39it/s]
[2026-10-17 15:21:02] Capture target verify CUDA graph end. bs=[1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 40, 44, 48], elapsed=2.97 s, mem usage=0.81 GB, avail mem=30.35 GB.
[2026-10-17 15:21:02] Capture draft prefill CUDA graph begin. backend=breakable, num_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096, 4608, 5120, 5632, 6144, 6656, 7168, 7680, 8192], avail mem=30.35 GB

  0%|          | 0/58 [00:00<?, ?it/s]
Capturing prefill shape (num_tokens=8192 avail_mem=30.22 GB):   0%|          | 0/58 [00:00<?, ?it/s]
Capturing prefill shape (num_tokens=8192 avail_mem=30.22 GB):   2%|▏         | 1/58 [00:00<00:09,  6.12it/s]
Capturing prefill shape (num_tokens=7680 avail_mem=28.80 GB):   2%|▏         | 1/58 [00:00<00:09,  6.12it/s]
Capturing prefill shape (num_tokens=7168 avail_mem=28.43 GB):   2%|▏         | 1/58 [00:00<00:09,  6.12it/s]
Capturing prefill shape (num_tokens=7168 avail_mem=28.43 GB):   5%|▌         | 3/58 [00:00<00:05, 10.23it/s]
Capturing prefill shape (num_tokens=6656 avail_mem=28.67 GB):   5%|▌         | 3/58 [00:00<00:05, 10.23it/s]
Capturing prefill shape (num_tokens=6144 avail_mem=28.67 GB):   5%|▌         | 3/58 [00:00<00:05, 10.23it/s]
Capturing prefill shape (num_tokens=6144 avail_mem=28.67 GB):   9%|▊         | 5/58 [00:00<00:04, 11.92it/s]
Capturing prefill shape (num_tokens=5632 avail_mem=28.67 GB):   9%|▊         | 5/58 [00:00<00:04, 11.92it/s]
Capturing prefill shape (num_tokens=5120 avail_mem=28.67 GB):   9%|▊         | 5/58 [00:00<00:04, 11.92it/s]
Capturing prefill shape (num_tokens=5120 avail_mem=28.67 GB):  12%|█▏        | 7/58 [00:00<00:03, 14.37it/s]
Capturing prefill shape (num_tokens=4608 avail_mem=28.67 GB):  12%|█▏        | 7/58 [00:00<00:03, 14.37it/s]
Capturing prefill shape (num_tokens=4096 avail_mem=28.67 GB):  12%|█▏        | 7/58 [00:00<00:03, 14.37it/s]
Capturing prefill shape (num_tokens=3840 avail_mem=28.67 GB):  12%|█▏        | 7/58 [00:00<00:03, 14.37it/s]
Capturing prefill shape (num_tokens=3840 avail_mem=28.67 GB):  17%|█▋        | 10/58 [00:00<00:02, 18.28it/s]
Capturing prefill shape (num_tokens=3584 avail_mem=28.67 GB):  17%|█▋        | 10/58 [00:00<00:02, 18.28it/s]
Capturing prefill shape (num_tokens=3328 avail_mem=28.67 GB):  17%|█▋        | 10/58 [00:00<00:02, 18.28it/s]
Capturing prefill shape (num_tokens=3072 avail_mem=28.66 GB):  17%|█▋        | 10/58 [00:00<00:02, 18.28it/s]
Capturing prefill shape (num_tokens=2816 avail_mem=28.66 GB):  17%|█▋        | 10/58 [00:00<00:02, 18.28it/s]
Capturing prefill shape (num_tokens=2816 avail_mem=28.66 GB):  24%|██▍       | 14/58 [00:00<00:01, 23.70it/s]
Capturing prefill shape (num_tokens=2560 avail_mem=28.66 GB):  24%|██▍       | 14/58 [00:00<00:01, 23.70it/s]
Capturing prefill shape (num_tokens=2304 avail_mem=28.65 GB):  24%|██▍       | 14/58 [00:00<00:01, 23.70it/s]
Capturing prefill shape (num_tokens=2048 avail_mem=28.65 GB):  24%|██▍       | 14/58 [00:00<00:01, 23.70it/s]
Capturing prefill shape (num_tokens=1792 avail_mem=28.65 GB):  24%|██▍       | 14/58 [00:00<00:01, 23.70it/s]
Capturing prefill shape (num_tokens=1536 avail_mem=28.65 GB):  24%|██▍       | 14/58 [00:00<00:01, 23.70it/s]
Capturing prefill shape (num_tokens=1280 avail_mem=28.65 GB):  24%|██▍       | 14/58 [00:00<00:01, 23.70it/s]
Capturing prefill shape (num_tokens=1280 avail_mem=28.65 GB):  34%|███▍      | 20/58 [00:00<00:01, 33.75it/s]
Capturing prefill shape (num_tokens=1024 avail_mem=28.65 GB):  34%|███▍      | 20/58 [00:00<00:01, 33.75it/s]
Capturing prefill shape (num_tokens=960 avail_mem=28.65 GB):  34%|███▍      | 20/58 [00:00<00:01, 33.75it/s] 
Capturing prefill shape (num_tokens=896 avail_mem=28.65 GB):  34%|███▍      | 20/58 [00:00<00:01, 33.75it/s]
Capturing prefill shape (num_tokens=832 avail_mem=28.65 GB):  34%|███▍      | 20/58 [00:00<00:01, 33.75it/s]
Capturing prefill shape (num_tokens=768 avail_mem=28.65 GB):  34%|███▍      | 20/58 [00:00<00:01, 33.75it/s]
Capturing prefill shape (num_tokens=704 avail_mem=28.65 GB):  34%|███▍      | 20/58 [00:00<00:01, 33.75it/s]
Capturing prefill shape (num_tokens=640 avail_mem=28.65 GB):  34%|███▍      | 20/58 [00:00<00:01, 33.75it/s]
Capturing prefill shape (num_tokens=576 avail_mem=28.65 GB):  34%|███▍      | 20/58 [00:00<00:01, 33.75it/s]
Capturing prefill shape (num_tokens=512 avail_mem=28.65 GB):  34%|███▍      | 20/58 [00:00<00:01, 33.75it/s]
Capturing prefill shape (num_tokens=480 avail_mem=28.65 GB):  34%|███▍      | 20/58 [00:00<00:01, 33.75it/s]
Capturing prefill shape (num_tokens=448 avail_mem=28.65 GB):  34%|███▍      | 20/58 [00:00<00:01, 33.75it/s]
Capturing prefill shape (num_tokens=448 avail_mem=28.65 GB):  53%|█████▎    | 31/58 [00:00<00:00, 55.10it/s]
Capturing prefill shape (num_tokens=416 avail_mem=28.65 GB):  53%|█████▎    | 31/58 [00:00<00:00, 55.10it/s]
Capturing prefill shape (num_tokens=384 avail_mem=28.65 GB):  53%|█████▎    | 31/58 [00:01<00:00, 55.10it/s]
Capturing prefill shape (num_tokens=352 avail_mem=28.65 GB):  53%|█████▎    | 31/58 [00:01<00:00, 55.10it/s]
Capturing prefill shape (num_tokens=320 avail_mem=28.64 GB):  53%|█████▎    | 31/58 [00:01<00:00, 55.10it/s]
Capturing prefill shape (num_tokens=288 avail_mem=28.64 GB):  53%|█████▎    | 31/58 [00:01<00:00, 55.10it/s]
Capturing prefill shape (num_tokens=256 avail_mem=28.64 GB):  53%|█████▎    | 31/58 [00:01<00:00, 55.10it/s]
Capturing prefill shape (num_tokens=240 avail_mem=28.63 GB):  53%|█████▎    | 31/58 [00:01<00:00, 55.10it/s]
Capturing prefill shape (num_tokens=224 avail_mem=28.63 GB):  53%|█████▎    | 31/58 [00:01<00:00, 55.10it/s]
Capturing prefill shape (num_tokens=208 avail_mem=28.63 GB):  53%|█████▎    | 31/58 [00:01<00:00, 55.10it/s]
Capturing prefill shape (num_tokens=192 avail_mem=28.63 GB):  53%|█████▎    | 31/58 [00:01<00:00, 55.10it/s]
Capturing prefill shape (num_tokens=176 avail_mem=28.63 GB):  53%|█████▎    | 31/58 [00:01<00:00, 55.10it/s]
Capturing prefill shape (num_tokens=160 avail_mem=28.63 GB):  53%|█████▎    | 31/58 [00:01<00:00, 55.10it/s]
Capturing prefill shape (num_tokens=160 avail_mem=28.63 GB):  74%|███████▍  | 43/58 [00:01<00:00, 73.49it/s]
Capturing prefill shape (num_tokens=144 avail_mem=28.63 GB):  74%|███████▍  | 43/58 [00:01<00:00, 73.49it/s]
Capturing prefill shape (num_tokens=128 avail_mem=28.62 GB):  74%|███████▍  | 43/58 [00:01<00:00, 73.49it/s]
Capturing prefill shape (num_tokens=112 avail_mem=28.62 GB):  74%|███████▍  | 43/58 [00:01<00:00, 73.49it/s]
Capturing prefill shape (num_tokens=96 avail_mem=28.62 GB):  74%|███████▍  | 43/58 [00:01<00:00, 73.49it/s] 
Capturing prefill shape (num_tokens=80 avail_mem=28.62 GB):  74%|███████▍  | 43/58 [00:01<00:00, 73.49it/s]
Capturing prefill shape (num_tokens=64 avail_mem=28.62 GB):  74%|███████▍  | 43/58 [00:01<00:00, 73.49it/s]
Capturing prefill shape (num_tokens=48 avail_mem=28.63 GB):  74%|███████▍  | 43/58 [00:01<00:00, 73.49it/s]
Capturing prefill shape (num_tokens=32 avail_mem=28.63 GB):  74%|███████▍  | 43/58 [00:01<00:00, 73.49it/s]
Capturing prefill shape (num_tokens=28 avail_mem=28.62 GB):  74%|███████▍  | 43/58 [00:01<00:00, 73.49it/s]
Capturing prefill shape (num_tokens=24 avail_mem=28.62 GB):  74%|███████▍  | 43/58 [00:01<00:00, 73.49it/s]
Capturing prefill shape (num_tokens=20 avail_mem=28.62 GB):  74%|███████▍  | 43/58 [00:01<00:00, 73.49it/s]
Capturing prefill shape (num_tokens=16 avail_mem=28.62 GB):  74%|███████▍  | 43/58 [00:01<00:00, 73.49it/s]
Capturing prefill shape (num_tokens=12 avail_mem=28.62 GB):  74%|███████▍  | 43/58 [00:01<00:00, 73.49it/s]
Capturing prefill shape (num_tokens=8 avail_mem=28.62 GB):  74%|███████▍  | 43/58 [00:01<00:00, 73.49it/s] 
Capturing prefill shape (num_tokens=4 avail_mem=28.62 GB):  74%|███████▍  | 43/58 [00:01<00:00, 73.49it/s]
Capturing prefill shape (num_tokens=4 avail_mem=28.62 GB): 100%|██████████| 58/58 [00:01<00:00, 49.56it/s]
[2026-10-17 15:21:03] Capture draft prefill CUDA graph end. elapsed=1.68 s, mem usage=0.62 GB, avail mem=29.73 GB.
[2026-10-17 15:21:03] Capture draft decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 40, 44, 48], avail mem=29.73 GB

  0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=48 avail_mem=29.59 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=48 avail_mem=29.59 GB):   4%|▍         | 1/23 [00:01<00:30,  1.37s/it]
Capturing batches (bs=44 avail_mem=29.24 GB):   4%|▍         | 1/23 [00:01<00:30,  1.37s/it]
Capturing batches (bs=40 avail_mem=29.20 GB):   4%|▍         | 1/23 [00:01<00:30,  1.37s/it]
Capturing batches (bs=32 avail_mem=29.18 GB):   4%|▍         | 1/23 [00:01<00:30,  1.37s/it]
Capturing batches (bs=30 avail_mem=29.13 GB):   4%|▍         | 1/23 [00:01<00:30,  1.37s/it]
Capturing batches (bs=30 avail_mem=29.13 GB):  22%|██▏       | 5/23 [00:01<00:04,  4.35it/s]
Capturing batches (bs=28 avail_mem=29.10 GB):  22%|██▏       | 5/23 [00:01<00:04,  4.35it/s]
Capturing batches (bs=26 avail_mem=29.05 GB):  22%|██▏       | 5/23 [00:01<00:04,  4.35it/s]
Capturing batches (bs=24 avail_mem=29.01 GB):  22%|██▏       | 5/23 [00:01<00:04,  4.35it/s]
Capturing batches (bs=22 avail_mem=28.97 GB):  22%|██▏       | 5/23 [00:01<00:04,  4.35it/s]
Capturing batches (bs=20 avail_mem=28.93 GB):  22%|██▏       | 5/23 [00:01<00:04,  4.35it/s]
Capturing batches (bs=20 avail_mem=28.93 GB):  43%|████▎     | 10/23 [00:01<00:01,  9.55it/s]
Capturing batches (bs=18 avail_mem=28.89 GB):  43%|████▎     | 10/23 [00:01<00:01,  9.55it/s]
Capturing batches (bs=16 avail_mem=28.85 GB):  43%|████▎     | 10/23 [00:01<00:01,  9.55it/s]
Capturing batches (bs=14 avail_mem=28.81 GB):  43%|████▎     | 10/23 [00:01<00:01,  9.55it/s]
Capturing batches (bs=12 avail_mem=28.77 GB):  43%|████▎     | 10/23 [00:01<00:01,  9.55it/s]
Capturing batches (bs=10 avail_mem=28.74 GB):  43%|████▎     | 10/23 [00:01<00:01,  9.55it/s]
Capturing batches (bs=10 avail_mem=28.74 GB):  65%|██████▌   | 15/23 [00:01<00:00, 15.01it/s]
Capturing batches (bs=8 avail_mem=28.70 GB):  65%|██████▌   | 15/23 [00:01<00:00, 15.01it/s] 
Capturing batches (bs=7 avail_mem=28.65 GB):  65%|██████▌   | 15/23 [00:01<00:00, 15.01it/s]
Capturing batches (bs=6 avail_mem=28.61 GB):  65%|██████▌   | 15/23 [00:01<00:00, 15.01it/s]
Capturing batches (bs=5 avail_mem=28.83 GB):  65%|██████▌   | 15/23 [00:01<00:00, 15.01it/s]
Capturing batches (bs=4 avail_mem=28.79 GB):  65%|██████▌   | 15/23 [00:01<00:00, 15.01it/s]
Capturing batches (bs=4 avail_mem=28.79 GB):  87%|████████▋ | 20/23 [00:01<00:00, 20.16it/s]
Capturing batches (bs=3 avail_mem=28.74 GB):  87%|████████▋ | 20/23 [00:01<00:00, 20.16it/s]
Capturing batches (bs=2 avail_mem=28.71 GB):  87%|████████▋ | 20/23 [00:01<00:00, 20.16it/s]
Capturing batches (bs=1 avail_mem=28.62 GB):  87%|████████▋ | 20/23 [00:01<00:00, 20.16it/s]
Capturing batches (bs=1 avail_mem=28.62 GB): 100%|██████████| 23/23 [00:02<00:00, 11.10it/s]
[2026-10-17 15:21:06] Capture draft decode CUDA graph end. elapsed=2.61 s, mem usage=1.68 GB, avail mem=28.05 GB.
[2026-10-17 15:21:06] Capture draft extend CUDA graph begin. backend=full, num_tokens_per_req=4, bs=[1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 40, 44, 48], avail mem=28.05 GB

  0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=48 avail_mem=28.28 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=44 avail_mem=28.27 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=40 avail_mem=28.22 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=32 avail_mem=28.22 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=30 avail_mem=28.18 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=28 avail_mem=28.18 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=26 avail_mem=28.14 GB):   0%|          | 0/23 [00:00<?, ?it/s]
Capturing batches (bs=26 avail_mem=28.14 GB):  30%|███       | 7/23 [00:00<00:00, 64.02it/s]
Capturing batches (bs=24 avail_mem=28.14 GB):  30%|███       | 7/23 [00:00<00:00, 64.02it/s]
Capturing batches (bs=22 avail_mem=28.10 GB):  30%|███       | 7/23 [00:00<00:00, 64.02it/s]
Capturing batches (bs=20 avail_mem=28.10 GB):  30%|███       | 7/23 [00:00<00:00, 64.02it/s]
Capturing batches (bs=18 avail_mem=28.07 GB):  30%|███       | 7/23 [00:00<00:00, 64.02it/s]
Capturing batches (bs=16 avail_mem=28.05 GB):  30%|███       | 7/23 [00:00<00:00, 64.02it/s]
Capturing batches (bs=14 avail_mem=28.02 GB):  30%|███       | 7/23 [00:00<00:00, 64.02it/s]
Capturing batches (bs=12 avail_mem=28.01 GB):  30%|███       | 7/23 [00:00<00:00, 64.02it/s]
Capturing batches (bs=10 avail_mem=27.98 GB):  30%|███       | 7/23 [00:00<00:00, 64.02it/s]
Capturing batches (bs=10 avail_mem=27.98 GB):  65%|██████▌   | 15/23 [00:00<00:00, 70.75it/s]
Capturing batches (bs=8 avail_mem=27.98 GB):  65%|██████▌   | 15/23 [00:00<00:00, 70.75it/s] 
Capturing batches (bs=7 avail_mem=27.94 GB):  65%|██████▌   | 15/23 [00:00<00:00, 70.75it/s]
Capturing batches (bs=6 avail_mem=27.94 GB):  65%|██████▌   | 15/23 [00:00<00:00, 70.75it/s]
Capturing batches (bs=5 avail_mem=27.91 GB):  65%|██████▌   | 15/23 [00:00<00:00, 70.75it/s]
Capturing batches (bs=4 avail_mem=27.90 GB):  65%|██████▌   | 15/23 [00:00<00:00, 70.75it/s]
Capturing batches (bs=3 avail_mem=27.45 GB):  65%|██████▌   | 15/23 [00:00<00:00, 70.75it/s]
Capturing batches (bs=2 avail_mem=27.44 GB):  65%|██████▌   | 15/23 [00:00<00:00, 70.75it/s]
Capturing batches (bs=1 avail_mem=27.41 GB):  65%|██████▌   | 15/23 [00:00<00:00, 70.75it/s]
Capturing batches (bs=1 avail_mem=27.41 GB): 100%|██████████| 23/23 [00:00<00:00, 73.81it/s]
Capturing batches (bs=1 avail_mem=27.41 GB): 100%|██████████| 23/23 [00:00<00:00, 72.25it/s]
[2026-10-17 15:21:07] Capture draft extend CUDA graph end. elapsed=0.90 s, mem usage=0.33 GB, avail mem=27.72 GB.
[2026-10-17 15:21:07] max_total_num_tokens=216775, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=48, context_len=262144, available_gpu_mem=27.72 GB
[2026-10-17 15:21:07] Init Unified Radix Cache. Components: (<ComponentType.FULL: 0>,). Tree Core: RustUnifiedTreeCore
[2026-10-17 15:21:07] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=False hybrid_ssm=False hicache_attached=False
[2026-10-17 15:21:07] Engine startup timings (s): load_weight=313.69, kv_cache_allocation=2.43, scheduler_e2e=363.09, cuda_graph={prefill=34.46, decode=0.00, target_verify=2.97, draft_prefill=1.68, draft_decode=2.61, draft_extend=0.90}, tokenizer_e2e=367.83
[2026-10-17 15:21:07] INFO:     Started server process [126678]
[2026-10-17 15:21:07] INFO:     Waiting for application startup.
[2026-10-17 15:21:07] Using default chat sampling params from model generation config: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
[2026-10-17 15:21:07] INFO:     Application startup complete.
[2026-10-17 15:21:07] INFO:     Uvicorn running on http://127.0.0.1:30000 (Press CTRL+C to quit)
[2026-10-17 15:21:08] INFO:     127.0.0.1:38018 - "GET /health HTTP/1.1" 503 Service Unavailable
[2026-10-17 15:21:08] INFO:     127.0.0.1:38032 - "GET /model_info HTTP/1.1" 200 OK
[2026-10-17 15:21:08] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 3.65
[2026-10-17 15:21:09] INFO:     127.0.0.1:38044 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:21:09] INFO:     127.0.0.1:38046 - "POST /freeze_gc HTTP/1.1" 200 OK
[2026-10-17 15:21:09] The server is fired up and ready to roll!
[2026-10-17 15:21:10] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.59
[2026-10-17 15:21:11] INFO:     127.0.0.1:49776 - "GET /health HTTP/1.1" 200 OK
[2026-10-17 15:21:11] Prefill batch, #new-seq: 1, #new-token: 4, #cached-token: 1, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 3.83
[2026-10-17 15:21:12] INFO:     127.0.0.1:49782 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:21:12] Prefill batch, #new-seq: 1, #new-token: 17, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 26.43
[2026-10-17 15:21:12] INFO:     127.0.0.1:49788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
/home/rohit/sglang-lab/.venv/lib/python3.12/site-packages/sglang/srt/entrypoints/http_server.py:476: FastAPIDeprecationWarning: ORJSONResponse is deprecated, FastAPI now serializes data directly to JSON bytes via Pydantic when a return type or response model is set, which is faster and doesn't need a custom response class. Read more in the FastAPI docs: https://fastapi.tiangolo.com/advanced/custom-response/#orjson-or-response-model and https://fastapi.tiangolo.com/tutorial/response-model/
  return await original_handler(ORJSONRequest(request.scope, request.receive))
[2026-10-17 15:21:19] INFO:     127.0.0.1:40212 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-17 15:21:20] INFO:     127.0.0.1:40228 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-17 15:21:20] INFO:     127.0.0.1:40238 - "GET /model_info HTTP/1.1" 200 OK
[2026-10-17 15:21:24] INFO:     127.0.0.1:40252 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:21:25] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 39.39
[2026-10-17 15:21:27] INFO:     127.0.0.1:40262 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:21:27] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 0.98
[2026-10-17 15:21:28] Decode batch, #running-req: 1, #token: 549, token usage: 0.00, accept len: 2.30, accept rate: 0.43, cuda graph: True, gen throughput (token/s): 4.38, #queue-req: 0
[2026-10-17 15:21:30] INFO:     127.0.0.1:34146 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:21:30] Prefill batch, #new-seq: 1, #new-token: 256, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 85.60
[2026-10-17 15:21:30] Decode batch, #running-req: 1, #token: 274, token usage: 0.00, accept len: 2.73, accept rate: 0.57, cuda graph: True, gen throughput (token/s): 40.77, #queue-req: 0
[2026-10-17 15:21:33] Decode batch, #running-req: 1, #token: 314, token usage: 0.00, accept len: 1.00, accept rate: 0.00, cuda graph: True, gen throughput (token/s): 17.51, #queue-req: 0
[2026-10-17 15:21:35] Decode batch, #running-req: 1, #token: 354, token usage: 0.00, accept len: 1.00, accept rate: 0.00, cuda graph: True, gen throughput (token/s): 17.51, #queue-req: 0
[2026-10-17 15:21:37] INFO:     127.0.0.1:34150 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:21:37] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 68.38
[2026-10-17 15:21:38] Decode batch, #running-req: 1, #token: 527, token usage: 0.00, accept len: 1.18, accept rate: 0.06, cuda graph: True, gen throughput (token/s): 18.58, #queue-req: 0
[2026-10-17 15:21:40] INFO:     127.0.0.1:45854 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:21:40] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 205.93
[2026-10-17 15:21:40] Decode batch, #running-req: 1, #token: 523, token usage: 0.00, accept len: 3.10, accept rate: 0.70, cuda graph: True, gen throughput (token/s): 50.17, #queue-req: 0
[2026-10-17 15:21:42] Decode batch, #running-req: 1, #token: 563, token usage: 0.00, accept len: 1.00, accept rate: 0.00, cuda graph: True, gen throughput (token/s): 17.98, #queue-req: 0
[2026-10-17 15:21:45] Decode batch, #running-req: 1, #token: 608, token usage: 0.00, accept len: 1.12, accept rate: 0.04, cuda graph: True, gen throughput (token/s): 19.99, #queue-req: 0
[2026-10-17 15:21:46] INFO:     127.0.0.1:45870 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:21:46] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 77.79
[2026-10-17 15:21:47] Decode batch, #running-req: 1, #token: 554, token usage: 0.00, accept len: 1.93, accept rate: 0.31, cuda graph: True, gen throughput (token/s): 30.32, #queue-req: 0
[2026-10-17 15:21:48] INFO:     127.0.0.1:45882 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:21:49] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 225.72
[2026-10-17 15:21:50] Decode batch, #running-req: 1, #token: 572, token usage: 0.00, accept len: 3.70, accept rate: 0.90, cuda graph: True, gen throughput (token/s): 56.16, #queue-req: 0
[2026-10-17 15:21:51] INFO:     127.0.0.1:46480 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:21:52] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 165.49
[2026-10-17 15:21:52] Decode batch, #running-req: 1, #token: 537, token usage: 0.00, accept len: 2.35, accept rate: 0.45, cuda graph: True, gen throughput (token/s): 34.45, #queue-req: 0
[2026-10-17 15:21:55] Decode batch, #running-req: 1, #token: 611, token usage: 0.00, accept len: 1.85, accept rate: 0.28, cuda graph: True, gen throughput (token/s): 32.13, #queue-req: 0
[2026-10-17 15:21:56] INFO:     127.0.0.1:46494 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:21:56] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 119.84
[2026-10-17 15:21:57] Decode batch, #running-req: 1, #token: 599, token usage: 0.00, accept len: 3.02, accept rate: 0.68, cuda graph: True, gen throughput (token/s): 49.59, #queue-req: 0
[2026-10-17 15:21:58] INFO:     127.0.0.1:46508 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:21:58] Prefill batch, #new-seq: 1, #new-token: 497, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 243.13
[2026-10-17 15:22:00] Decode batch, #running-req: 1, #token: 610, token usage: 0.00, accept len: 3.92, accept rate: 0.97, cuda graph: True, gen throughput (token/s): 62.45, #queue-req: 0
[2026-10-17 15:22:00] INFO:     127.0.0.1:56098 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:00] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 231.36
[2026-10-17 15:22:03] Decode batch, #running-req: 1, #token: 613, token usage: 0.00, accept len: 2.98, accept rate: 0.66, cuda graph: True, gen throughput (token/s): 38.86, #queue-req: 0
[2026-10-17 15:22:04] INFO:     127.0.0.1:56102 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:04] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 123.50
[2026-10-17 15:22:05] Decode batch, #running-req: 1, #token: 582, token usage: 0.00, accept len: 2.55, accept rate: 0.52, cuda graph: True, gen throughput (token/s): 38.03, #queue-req: 0
[2026-10-17 15:22:06] INFO:     127.0.0.1:56118 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:07] Prefill batch, #new-seq: 1, #new-token: 509, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 233.45
[2026-10-17 15:22:08] Decode batch, #running-req: 1, #token: 564, token usage: 0.00, accept len: 2.83, accept rate: 0.61, cuda graph: True, gen throughput (token/s): 44.21, #queue-req: 0
[2026-10-17 15:22:10] Decode batch, #running-req: 1, #token: 0, token usage: 0.00, accept len: 2.00, accept rate: 0.33, cuda graph: True, gen throughput (token/s): 35.14, #queue-req: 0
[2026-10-17 15:22:10] INFO:     127.0.0.1:56530 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:11] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 130.73
[2026-10-17 15:22:13] INFO:     127.0.0.1:56546 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:13] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 214.04
[2026-10-17 15:22:13] Decode batch, #running-req: 1, #token: 522, token usage: 0.00, accept len: 3.40, accept rate: 0.80, cuda graph: True, gen throughput (token/s): 48.89, #queue-req: 0
[2026-10-17 15:22:15] Decode batch, #running-req: 1, #token: 586, token usage: 0.00, accept len: 1.60, accept rate: 0.20, cuda graph: True, gen throughput (token/s): 27.95, #queue-req: 0
[2026-10-17 15:22:17] INFO:     127.0.0.1:56562 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:18] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 109.66
[2026-10-17 15:22:18] Decode batch, #running-req: 1, #token: 540, token usage: 0.00, accept len: 2.17, accept rate: 0.39, cuda graph: True, gen throughput (token/s): 32.98, #queue-req: 0
[2026-10-17 15:22:20] INFO:     127.0.0.1:38516 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:20] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 221.68
[2026-10-17 15:22:21] Decode batch, #running-req: 1, #token: 559, token usage: 0.00, accept len: 3.77, accept rate: 0.93, cuda graph: True, gen throughput (token/s): 57.05, #queue-req: 0
[2026-10-17 15:22:22] INFO:     127.0.0.1:38526 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-17 15:22:22] INFO:     127.0.0.1:38532 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-17 15:22:29] INFO:     127.0.0.1:38546 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-17 15:22:29] INFO:     127.0.0.1:38552 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-17 15:22:29] INFO:     127.0.0.1:38564 - "GET /model_info HTTP/1.1" 200 OK
[2026-10-17 15:22:33] INFO:     127.0.0.1:59214 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:33] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 0.15
[2026-10-17 15:22:35] INFO:     127.0.0.1:59218 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:35] INFO:     127.0.0.1:59234 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:35] INFO:     127.0.0.1:59240 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:35] INFO:     127.0.0.1:59252 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:35] INFO:     127.0.0.1:59268 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:35] INFO:     127.0.0.1:59278 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:35] INFO:     127.0.0.1:59292 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:35] INFO:     127.0.0.1:59300 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:35] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1.02
[2026-10-17 15:22:35] Prefill batch, #new-seq: 7, #new-token: 14, #cached-token: 3314, token usage: 0.02, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 75.91
[2026-10-17 15:22:36] Decode batch, #running-req: 8, #token: 3966, token usage: 0.02, accept len: 3.03, accept rate: 0.68, cuda graph: True, gen throughput (token/s): 13.46, #queue-req: 0
[2026-10-17 15:22:41] INFO:     127.0.0.1:48012 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:41] INFO:     127.0.0.1:48018 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:41] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 495, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 0.35
[2026-10-17 15:22:41] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 509, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 8.93
[2026-10-17 15:22:42] INFO:     127.0.0.1:48024 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:42] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1.78
[2026-10-17 15:22:43] INFO:     127.0.0.1:48038 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:43] Decode batch, #running-req: 7, #token: 4215, token usage: 0.02, accept len: 2.57, accept rate: 0.52, cuda graph: True, gen throughput (token/s): 114.37, #queue-req: 0
[2026-10-17 15:22:43] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 507, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 3.40
[2026-10-17 15:22:43] INFO:     127.0.0.1:48046 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:44] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 3.44
[2026-10-17 15:22:47] INFO:     127.0.0.1:48054 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:48] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 0.51
[2026-10-17 15:22:48] INFO:     127.0.0.1:48070 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:48] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 2.68
[2026-10-17 15:22:48] INFO:     127.0.0.1:48080 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:49] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 3.60
[2026-10-17 15:22:50] Decode batch, #running-req: 8, #token: 3821, token usage: 0.02, accept len: 2.33, accept rate: 0.44, cuda graph: True, gen throughput (token/s): 102.15, #queue-req: 0
[2026-10-17 15:22:50] INFO:     127.0.0.1:57508 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:51] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 255.63
[2026-10-17 15:22:52] INFO:     127.0.0.1:57518 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:52] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 335.00
[2026-10-17 15:22:55] INFO:     127.0.0.1:57528 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:55] INFO:     127.0.0.1:57544 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:55] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 181.15
[2026-10-17 15:22:56] Prefill batch, #new-seq: 1, #new-token: 507, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 816.75
[2026-10-17 15:22:56] INFO:     127.0.0.1:57554 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:57] Prefill batch, #new-seq: 1, #new-token: 488, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 504.45
[2026-10-17 15:22:59] INFO:     127.0.0.1:57562 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:22:59] Decode batch, #running-req: 7, #token: 3940, token usage: 0.02, accept len: 2.58, accept rate: 0.53, cuda graph: True, gen throughput (token/s): 93.41, #queue-req: 0
[2026-10-17 15:22:59] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 214.02
[2026-10-17 15:22:59] INFO:     127.0.0.1:53500 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:00] INFO:     127.0.0.1:53510 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:00] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 6, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 501.95
[2026-10-17 15:23:00] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 70693.84
[2026-10-17 15:23:01] INFO:     127.0.0.1:53520 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:01] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 529.78
[2026-10-17 15:23:03] INFO:     127.0.0.1:53524 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:03] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 230.59
[2026-10-17 15:23:05] INFO:     127.0.0.1:53534 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:06] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 1, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 181.51
[2026-10-17 15:23:06] INFO:     127.0.0.1:53546 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:06] INFO:     127.0.0.1:53550 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:07] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 6, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 604.08
[2026-10-17 15:23:07] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 71310.02
[2026-10-17 15:23:08] Decode batch, #running-req: 8, #token: 4074, token usage: 0.02, accept len: 2.86, accept rate: 0.62, cuda graph: True, gen throughput (token/s): 92.77, #queue-req: 0
[2026-10-17 15:23:08] INFO:     127.0.0.1:53554 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:09] INFO:     127.0.0.1:53568 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:09] Prefill batch, #new-seq: 1, #new-token: 509, #cached-token: 3, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 272.38
[2026-10-17 15:23:09] INFO:     127.0.0.1:53580 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:10] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 553.56
[2026-10-17 15:23:10] Prefill batch, #new-seq: 1, #new-token: 510, #cached-token: 0, token usage: 0.02, #running-req: 8, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 67450.48
[2026-10-17 15:23:13] INFO:     127.0.0.1:48062 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:13] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 152.19
[2026-10-17 15:23:15] INFO:     127.0.0.1:48078 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:15] INFO:     127.0.0.1:48090 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:16] Prefill batch, #new-seq: 2, #new-token: 1020, #cached-token: 0, token usage: 0.02, #running-req: 6, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 421.59
[2026-10-17 15:23:16] INFO:     127.0.0.1:48102 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:17] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 513.78
[2026-10-17 15:23:18] Decode batch, #running-req: 8, #token: 4708, token usage: 0.02, accept len: 2.88, accept rate: 0.63, cuda graph: True, gen throughput (token/s): 96.78, #queue-req: 0
[2026-10-17 15:23:18] INFO:     127.0.0.1:48104 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:18] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 299.72
[2026-10-17 15:23:19] INFO:     127.0.0.1:48112 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:19] INFO:     127.0.0.1:48114 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:19] Prefill batch, #new-seq: 2, #new-token: 1024, #cached-token: 0, token usage: 0.02, #running-req: 6, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1235.65
[2026-10-17 15:23:21] INFO:     127.0.0.1:33534 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:22] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 185.51
[2026-10-17 15:23:23] INFO:     127.0.0.1:33550 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:24] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 268.75
[2026-10-17 15:23:24] INFO:     127.0.0.1:33562 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:24] INFO:     127.0.0.1:33574 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:25] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 483.17
[2026-10-17 15:23:25] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.02, #running-req: 8, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 71853.69
[2026-10-17 15:23:26] Decode batch, #running-req: 8, #token: 4674, token usage: 0.02, accept len: 2.80, accept rate: 0.60, cuda graph: True, gen throughput (token/s): 100.46, #queue-req: 0
[2026-10-17 15:23:27] INFO:     127.0.0.1:33590 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:28] INFO:     127.0.0.1:33598 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:28] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 191.95
[2026-10-17 15:23:28] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 838.61
[2026-10-17 15:23:30] INFO:     127.0.0.1:39670 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:30] INFO:     127.0.0.1:39680 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:30] INFO:     127.0.0.1:39682 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:31] Prefill batch, #new-seq: 3, #new-token: 1753, #cached-token: 0, token usage: 0.02, #running-req: 5, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 580.45
[2026-10-17 15:23:32] INFO:     127.0.0.1:39694 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:32] INFO:     127.0.0.1:39708 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:32] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 556.88
[2026-10-17 15:23:33] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 856.33
[2026-10-17 15:23:35] Decode batch, #running-req: 8, #token: 4853, token usage: 0.02, accept len: 2.78, accept rate: 0.59, cuda graph: True, gen throughput (token/s): 102.01, #queue-req: 0
[2026-10-17 15:23:35] INFO:     127.0.0.1:39710 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:36] INFO:     127.0.0.1:39724 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:36] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 144.79
[2026-10-17 15:23:36] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.02, #running-req: 8, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 72269.09
[2026-10-17 15:23:39] INFO:     127.0.0.1:39738 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:39] INFO:     127.0.0.1:39742 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:39] INFO:     127.0.0.1:58340 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:40] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 149.03
[2026-10-17 15:23:40] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 8, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1852.75
[2026-10-17 15:23:40] Prefill batch, #new-seq: 1, #new-token: 510, #cached-token: 0, token usage: 0.02, #running-req: 9, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 68770.15
[2026-10-17 15:23:41] INFO:     127.0.0.1:58344 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:42] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 317.89
[2026-10-17 15:23:44] Decode batch, #running-req: 8, #token: 4035, token usage: 0.02, accept len: 2.65, accept rate: 0.55, cuda graph: True, gen throughput (token/s): 93.06, #queue-req: 0
[2026-10-17 15:23:44] INFO:     127.0.0.1:58346 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:44] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 187.16
[2026-10-17 15:23:47] INFO:     127.0.0.1:58362 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:48] Prefill batch, #new-seq: 1, #new-token: 493, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 151.42
[2026-10-17 15:23:49] INFO:     127.0.0.1:58364 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:49] INFO:     127.0.0.1:58376 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:49] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 309.00
[2026-10-17 15:23:50] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 803.37
[2026-10-17 15:23:52] INFO:     127.0.0.1:45006 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:52] Decode batch, #running-req: 8, #token: 4258, token usage: 0.02, accept len: 2.41, accept rate: 0.47, cuda graph: True, gen throughput (token/s): 86.56, #queue-req: 0
[2026-10-17 15:23:53] INFO:     127.0.0.1:45014 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:53] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 1, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 172.43
[2026-10-17 15:23:53] INFO:     127.0.0.1:45028 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:53] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 843.69
[2026-10-17 15:23:54] Prefill batch, #new-seq: 1, #new-token: 342, #cached-token: 0, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1153.25
[2026-10-17 15:23:55] INFO:     127.0.0.1:45044 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:23:55] Prefill batch, #new-seq: 1, #new-token: 508, #cached-token: 1, token usage: 0.02, #running-req: 7, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 316.06
[2026-10-17 15:23:59] Decode batch, #running-req: 3, #token: 1214, token usage: 0.01, accept len: 3.15, accept rate: 0.72, cuda graph: True, gen throughput (token/s): 108.22, #queue-req: 0
[2026-10-17 15:24:01] INFO:     127.0.0.1:50624 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-17 15:24:01] INFO:     127.0.0.1:50628 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-17 15:24:07] INFO:     127.0.0.1:50630 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-17 15:24:07] INFO:     127.0.0.1:50644 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-17 15:24:07] INFO:     127.0.0.1:50648 - "GET /model_info HTTP/1.1" 200 OK
[2026-10-17 15:24:11] INFO:     127.0.0.1:38896 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:11] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 0.13
[2026-10-17 15:24:12] Decode batch, #running-req: 1, #token: 0, token usage: 0.00, accept len: 2.46, accept rate: 0.49, cuda graph: True, gen throughput (token/s): 8.57, #queue-req: 0
[2026-10-17 15:24:13] INFO:     127.0.0.1:38904 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:38918 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:38920 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:38934 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:38948 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:38960 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:38972 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:38986 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:38988 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:38994 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39010 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39020 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39026 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39034 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39040 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39046 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39052 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39064 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39068 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39078 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39094 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39098 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39104 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39118 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39126 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39136 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39140 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39150 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39166 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39178 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39182 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] INFO:     127.0.0.1:39186 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:13] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.07, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 0.96
[2026-10-17 15:24:14] Prefill batch, #new-seq: 31, #new-token: 62, #cached-token: 15502, token usage: 0.08, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 220.33
[2026-10-17 15:24:22] INFO:     127.0.0.1:50468 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:22] INFO:     127.0.0.1:50472 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:22] INFO:     127.0.0.1:50484 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:22] INFO:     127.0.0.1:50488 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:22] INFO:     127.0.0.1:50496 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:22] INFO:     127.0.0.1:50508 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:22] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 509, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 0.23
[2026-10-17 15:24:22] Prefill batch, #new-seq: 3, #new-token: 6, #cached-token: 1526, token usage: 0.09, #running-req: 32, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 93.14
[2026-10-17 15:24:22] Prefill batch, #new-seq: 2, #new-token: 4, #cached-token: 1020, token usage: 0.09, #running-req: 35, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 591.60
[2026-10-17 15:24:23] INFO:     127.0.0.1:50510 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:23] INFO:     127.0.0.1:50516 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:23] INFO:     127.0.0.1:50528 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:23] INFO:     127.0.0.1:50536 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:23] INFO:     127.0.0.1:50538 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:23] INFO:     127.0.0.1:50542 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:23] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.08, #running-req: 30, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 2.15
[2026-10-17 15:24:23] Prefill batch, #new-seq: 3, #new-token: 6, #cached-token: 1529, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 104.11
[2026-10-17 15:24:23] Prefill batch, #new-seq: 2, #new-token: 4, #cached-token: 1019, token usage: 0.08, #running-req: 34, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 637.59
[2026-10-17 15:24:24] INFO:     127.0.0.1:50558 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:24] INFO:     127.0.0.1:50572 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:24] INFO:     127.0.0.1:50584 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:24] INFO:     127.0.0.1:50590 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:24] INFO:     127.0.0.1:50596 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:24] Decode batch, #running-req: 30, #token: 14277, token usage: 0.07, accept len: 2.96, accept rate: 0.65, cuda graph: True, gen throughput (token/s): 310.16, #queue-req: 0
[2026-10-17 15:24:24] INFO:     127.0.0.1:50612 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:24] INFO:     127.0.0.1:50618 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:24] INFO:     127.0.0.1:50628 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:24] Prefill batch, #new-seq: 2, #new-token: 4, #cached-token: 1019, token usage: 0.07, #running-req: 30, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 4.28
[2026-10-17 15:24:24] Prefill batch, #new-seq: 3, #new-token: 6, #cached-token: 1748, token usage: 0.08, #running-req: 32, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 65.92
[2026-10-17 15:24:24] Prefill batch, #new-seq: 3, #new-token: 6, #cached-token: 1529, token usage: 0.08, #running-req: 35, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 903.11
[2026-10-17 15:24:25] INFO:     127.0.0.1:50640 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:25] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 509, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1.82
[2026-10-17 15:24:26] INFO:     127.0.0.1:50656 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:26] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 2.37
[2026-10-17 15:24:27] INFO:     127.0.0.1:50672 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:28] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 508, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1.21
[2026-10-17 15:24:29] INFO:     127.0.0.1:50676 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:29] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 509, token usage: 0.09, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1.43
[2026-10-17 15:24:30] INFO:     127.0.0.1:49778 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:31] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 509, token usage: 0.09, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1.20
[2026-10-17 15:24:31] INFO:     127.0.0.1:49790 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:31] INFO:     127.0.0.1:49792 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:32] INFO:     127.0.0.1:49796 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:32] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 491, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 3.13
[2026-10-17 15:24:32] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.09, #running-req: 32, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 34.22
[2026-10-17 15:24:32] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.09, #running-req: 33, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 312.57
[2026-10-17 15:24:32] INFO:     127.0.0.1:49808 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:33] INFO:     127.0.0.1:49824 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:33] INFO:     127.0.0.1:49832 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:33] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1.72
[2026-10-17 15:24:33] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 510, token usage: 0.09, #running-req: 32, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 35.96
[2026-10-17 15:24:33] Prefill batch, #new-seq: 1, #new-token: 2, #cached-token: 340, token usage: 0.09, #running-req: 33, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 327.11
[2026-10-17 15:24:33] INFO:     127.0.0.1:49840 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:33] INFO:     127.0.0.1:49856 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:33] INFO:     127.0.0.1:49860 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:33] INFO:     127.0.0.1:49868 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:34] INFO:     127.0.0.1:49882 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:34] INFO:     127.0.0.1:49886 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:35] Prefill batch, #new-seq: 4, #new-token: 1537, #cached-token: 508, token usage: 0.08, #running-req: 28, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 869.44
[2026-10-17 15:24:35] Prefill batch, #new-seq: 1, #new-token: 510, #cached-token: 1, token usage: 0.08, #running-req: 32, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1540.36
[2026-10-17 15:24:35] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.08, #running-req: 33, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 42010.15
[2026-10-17 15:24:35] INFO:     127.0.0.1:49900 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:36] INFO:     127.0.0.1:49906 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:36] INFO:     127.0.0.1:49914 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:37] INFO:     127.0.0.1:49928 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:37] Prefill batch, #new-seq: 1, #new-token: 510, #cached-token: 2, token usage: 0.08, #running-req: 30, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 254.29
[2026-10-17 15:24:38] Prefill batch, #new-seq: 2, #new-token: 1021, #cached-token: 2, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 2205.23
[2026-10-17 15:24:38] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.08, #running-req: 33, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 51197.48
[2026-10-17 15:24:38] INFO:     127.0.0.1:49940 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:38] INFO:     127.0.0.1:49950 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:38] INFO:     127.0.0.1:49958 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:39] Decode batch, #running-req: 29, #token: 16902, token usage: 0.08, accept len: 2.68, accept rate: 0.56, cuda graph: True, gen throughput (token/s): 228.72, #queue-req: 0
[2026-10-17 15:24:39] Prefill batch, #new-seq: 2, #new-token: 1023, #cached-token: 1, token usage: 0.08, #running-req: 29, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 540.75
[2026-10-17 15:24:39] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 40373.23
[2026-10-17 15:24:41] INFO:     127.0.0.1:57636 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:42] Prefill batch, #new-seq: 1, #new-token: 477, #cached-token: 1, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 197.12
[2026-10-17 15:24:43] INFO:     127.0.0.1:57638 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:44] Prefill batch, #new-seq: 1, #new-token: 510, #cached-token: 2, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 198.66
[2026-10-17 15:24:44] INFO:     127.0.0.1:57650 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:45] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 628.23
[2026-10-17 15:24:46] INFO:     127.0.0.1:57652 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:47] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.09, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 402.94
[2026-10-17 15:24:47] INFO:     127.0.0.1:57654 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:47] INFO:     127.0.0.1:57658 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:48] INFO:     127.0.0.1:57664 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:48] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 404.24
[2026-10-17 15:24:48] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.08, #running-req: 32, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1728.42
[2026-10-17 15:24:48] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.09, #running-req: 33, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 63793.48
[2026-10-17 15:24:48] INFO:     127.0.0.1:57674 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:48] INFO:     127.0.0.1:57676 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:49] INFO:     127.0.0.1:57686 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:50] Prefill batch, #new-seq: 2, #new-token: 1020, #cached-token: 4, token usage: 0.08, #running-req: 30, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 710.67
[2026-10-17 15:24:50] Prefill batch, #new-seq: 1, #new-token: 509, #cached-token: 3, token usage: 0.09, #running-req: 32, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 69154.40
[2026-10-17 15:24:50] INFO:     127.0.0.1:50558 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:51] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 370.81
[2026-10-17 15:24:51] INFO:     127.0.0.1:50574 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:51] INFO:     127.0.0.1:50580 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:52] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.09, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 498.53
[2026-10-17 15:24:52] Prefill batch, #new-seq: 1, #new-token: 500, #cached-token: 3, token usage: 0.09, #running-req: 32, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 69330.51
[2026-10-17 15:24:52] INFO:     127.0.0.1:50582 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:52] INFO:     127.0.0.1:50594 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:52] INFO:     127.0.0.1:50610 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:52] INFO:     127.0.0.1:50620 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:53] INFO:     127.0.0.1:50626 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:53] Prefill batch, #new-seq: 2, #new-token: 1012, #cached-token: 6, token usage: 0.08, #running-req: 30, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 672.81
[2026-10-17 15:24:54] Prefill batch, #new-seq: 2, #new-token: 1023, #cached-token: 1, token usage: 0.08, #running-req: 32, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 4410.97
[2026-10-17 15:24:54] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.08, #running-req: 34, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 68777.08
[2026-10-17 15:24:54] INFO:     127.0.0.1:50638 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:54] INFO:     127.0.0.1:50642 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:55] INFO:     127.0.0.1:50654 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:55] INFO:     127.0.0.1:50662 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:55] INFO:     127.0.0.1:50670 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:55] Prefill batch, #new-seq: 1, #new-token: 509, #cached-token: 3, token usage: 0.07, #running-req: 30, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 378.17
[2026-10-17 15:24:55] Prefill batch, #new-seq: 1, #new-token: 498, #cached-token: 1, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1234.77
[2026-10-17 15:24:55] Prefill batch, #new-seq: 3, #new-token: 1502, #cached-token: 0, token usage: 0.08, #running-req: 32, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 134327.84
[2026-10-17 15:24:57] Decode batch, #running-req: 32, #token: 17996, token usage: 0.08, accept len: 2.67, accept rate: 0.56, cuda graph: True, gen throughput (token/s): 191.82, #queue-req: 0
[2026-10-17 15:24:57] INFO:     127.0.0.1:50684 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:58] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 1, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 238.74
[2026-10-17 15:24:58] INFO:     127.0.0.1:50692 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:59] Prefill batch, #new-seq: 1, #new-token: 510, #cached-token: 2, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 479.37
[2026-10-17 15:24:59] INFO:     127.0.0.1:50694 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:59] INFO:     127.0.0.1:50706 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:24:59] INFO:     127.0.0.1:50708 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:00] Prefill batch, #new-seq: 3, #new-token: 1532, #cached-token: 4, token usage: 0.09, #running-req: 29, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1258.15
[2026-10-17 15:25:00] INFO:     127.0.0.1:59430 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:01] Prefill batch, #new-seq: 1, #new-token: 505, #cached-token: 0, token usage: 0.09, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 474.38
[2026-10-17 15:25:01] INFO:     127.0.0.1:59432 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:01] INFO:     127.0.0.1:59438 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:02] Prefill batch, #new-seq: 2, #new-token: 1023, #cached-token: 1, token usage: 0.09, #running-req: 30, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 755.47
[2026-10-17 15:25:03] INFO:     127.0.0.1:59454 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:03] INFO:     127.0.0.1:59462 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:04] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.09, #running-req: 30, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 386.43
[2026-10-17 15:25:04] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.09, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 68912.28
[2026-10-17 15:25:04] INFO:     127.0.0.1:59464 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:04] INFO:     127.0.0.1:59480 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:04] INFO:     127.0.0.1:59496 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:05] INFO:     127.0.0.1:59504 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:05] Prefill batch, #new-seq: 2, #new-token: 1018, #cached-token: 6, token usage: 0.08, #running-req: 30, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 601.02
[2026-10-17 15:25:06] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.09, #running-req: 32, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1701.81
[2026-10-17 15:25:06] Prefill batch, #new-seq: 1, #new-token: 510, #cached-token: 2, token usage: 0.09, #running-req: 33, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 66997.61
[2026-10-17 15:25:06] INFO:     127.0.0.1:59508 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:06] INFO:     127.0.0.1:59518 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:07] Prefill batch, #new-seq: 2, #new-token: 1022, #cached-token: 2, token usage: 0.09, #running-req: 30, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 712.07
[2026-10-17 15:25:07] INFO:     127.0.0.1:59522 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:07] INFO:     127.0.0.1:59526 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:08] INFO:     127.0.0.1:59534 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:08] INFO:     127.0.0.1:59536 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:08] INFO:     127.0.0.1:59542 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:09] Prefill batch, #new-seq: 2, #new-token: 1023, #cached-token: 1, token usage: 0.08, #running-req: 30, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 693.62
[2026-10-17 15:25:09] Prefill batch, #new-seq: 2, #new-token: 1023, #cached-token: 1, token usage: 0.08, #running-req: 32, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 3642.24
[2026-10-17 15:25:09] Prefill batch, #new-seq: 1, #new-token: 508, #cached-token: 4, token usage: 0.08, #running-req: 34, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 68568.07
[2026-10-17 15:25:09] INFO:     127.0.0.1:59556 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:09] INFO:     127.0.0.1:59564 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:09] INFO:     127.0.0.1:59570 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:10] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.08, #running-req: 30, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 372.45
[2026-10-17 15:25:10] Prefill batch, #new-seq: 2, #new-token: 1024, #cached-token: 0, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 116145.44
[2026-10-17 15:25:10] INFO:     127.0.0.1:40888 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:11] Prefill batch, #new-seq: 1, #new-token: 509, #cached-token: 0, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 473.64
[2026-10-17 15:25:11] INFO:     127.0.0.1:40900 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:12] Prefill batch, #new-seq: 1, #new-token: 509, #cached-token: 3, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 641.20
[2026-10-17 15:25:13] Decode batch, #running-req: 32, #token: 17866, token usage: 0.08, accept len: 2.84, accept rate: 0.61, cuda graph: True, gen throughput (token/s): 221.87, #queue-req: 0
[2026-10-17 15:25:13] INFO:     127.0.0.1:40910 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:13] Prefill batch, #new-seq: 1, #new-token: 510, #cached-token: 2, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 380.28
[2026-10-17 15:25:13] INFO:     127.0.0.1:40920 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:14] INFO:     127.0.0.1:40932 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:25:14] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.08, #running-req: 31, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 462.88
[2026-10-17 15:25:15] Prefill batch, #new-seq: 1, #new-token: 508, #cached-token: 4, token usage: 0.08, #running-req: 32, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 69212.44
[2026-10-17 15:25:24] Decode batch, #running-req: 9, #token: 4310, token usage: 0.02, accept len: 2.69, accept rate: 0.56, cuda graph: True, gen throughput (token/s): 205.60, #queue-req: 0
[2026-10-17 15:25:27] INFO:     127.0.0.1:39400 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-17 15:25:27] INFO:     127.0.0.1:39410 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-17 15:25:29] INFO:     127.0.0.1:39412 - "GET /metrics HTTP/1.1" 200 OK
[2026-10-17 15:25:29] SIGTERM received. signum=None frame=None. Draining requests and shutting down...
[2026-10-17 15:25:29] Subprocess detokenizer (pid=126770) crashed with exit code -15. Triggering SIGQUIT for cleanup...
[2026-10-17 15:25:29] SIGQUIT received. signum=None, frame=None. It usually means one child failed.
[2026-10-17 15:25:29] Sleeping 5 seconds before crash diagnostics to let GPU activity settle.
[2026-10-17 15:25:34] No live scheduler processes found; skipping py-spy and CUDA coredump.
[2026-10-17 15:25:34] kill_process_tree called: parent_pid=126678, include_parent=True, pid=126678