benchmark_args=Namespace(backend='sglang', base_url=None, host='0.0.0.0', port=30000, ready_check_timeout_sec=60, dataset_name='generated-shared-prefix', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model=None, served_model_name=None, tokenizer=None, num_prompts=1000, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=1024, random_output_len=1024, random_range_ratio=0.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=8, output_file='/private/tmp/sglang-research/capture/out/bench-shared-prefix.jsonl', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=7, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=False, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=4, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=64, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None, rust_client=False, rust_worker_threads=None) Waiting up to 60s for http://0.0.0.0:30000/v1/models to become ready... Server ready in 0.0s. Namespace(backend='sglang', base_url=None, host='0.0.0.0', port=30000, ready_check_timeout_sec=60, dataset_name='generated-shared-prefix', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model='mlx-community/Qwen3-0.6B-4bit', served_model_name=None, tokenizer=None, num_prompts=1000, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=1024, random_output_len=1024, random_range_ratio=0.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=8, output_file='/private/tmp/sglang-research/capture/out/bench-shared-prefix.jsonl', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=7, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=False, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=4, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=64, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None, rust_client=False, rust_worker_threads=None) Loading cached generated input data from ~/.cache/sglang/benchmark/gen_shared_prefix_7_4_16_2048_128_64_Qwen2Tokenizer.pkl Starting warmup with 1 sequences... Warmup completed with 1 sequences. Starting main benchmark run... ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 8 Successful requests: 64 Benchmark duration (s): 48.81 Total input tokens: 146646 Total input text tokens: 146646 Total generated tokens: 4096 Total generated tokens (retokenized): 2957 Request throughput (req/s): 1.31 Input token throughput (tok/s): 3004.13 Output token throughput (tok/s): 83.91 Peak output token throughput (tok/s): 160.00 Peak concurrent requests: 16 Total token throughput (tok/s): 3088.04 Concurrency: 7.93 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6047.52 Median E2E Latency (ms): 6191.72 P90 E2E Latency (ms): 6681.97 P95 E2E Latency (ms): 6684.90 P99 E2E Latency (ms): 6686.18 ---------------Time to First Token---------------- Mean TTFT (ms): 1284.35 Median TTFT (ms): 1358.02 P90 TTFT (ms): 1518.67 P95 TTFT (ms): 3152.28 P99 TTFT (ms): 3277.18 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 75.61 Median TPOT (ms): 80.79 P90 TPOT (ms): 86.63 P95 TPOT (ms): 93.66 P99 TPOT (ms): 99.15 ---------------Inter-Token Latency---------------- Mean ITL (ms): 75.59 Median ITL (ms): 65.53 P90 ITL (ms): 95.72 P95 ITL (ms): 127.60 P99 ITL (ms): 210.04 Max ITL (ms): 3040.81 ================================================== /private/tmp/sglang-research/sglang-v0.5.22/python/sglang/srt/layers/quantization/awq/awq.py:53: UserWarning: Only CUDA, HIP and XPU support AWQ currently. warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.") /private/tmp/sglang-research/sglang-v0.5.22/python/sglang/srt/layers/quantization/gguf.py:69: UserWarning: Only CUDA, MUSA and NPU support GGUF quantization currently. warnings.warn(f"Only CUDA, MUSA and NPU support GGUF quantization currently.") 0%| | 0/64 [00:00