ERR_SSL_Protocol_error in 1.8.11 with Vless-tls

[sky-chen]

Integrity requirements

• I confirm that I have read the documentation, understand the meaning of all the configuration items I wrote, and did not pile up seemingly useful options or default values.
• I searched issues and did not find any similar issues.

Version

1.8.11

Description

I followed official tutorial, both server and client config pretty much mirrors the tutorial. It's vless-tls with flow = "xtls-rprx-vision".

1. Verify xray and fallback running ok: I can access my site via https
2. But I can't use it normally as a tunnel, google, twitter can not be open. I'm getting ERR_SSL_Protocol_error, and a download page will be opened.
3. Tried both mac and window client, same result

I don't see suspicious log in server side, here's log from client:
xray.log

Reproduction Method

basically follow the tutorial with client and server 1.8.11

Client config

Server config

Client log

2024/05/16 22:05:41 [Info] [29082988] app/proxyman/inbound: connection ends > proxy/socks: connection ends > context canceled
2024/05/16 22:05:41 [Info] [3130882421] proxy: XtlsPadding 7 1357 0
2024/05/16 22:05:41 [Info] [3569570120] proxy/socks: TCP Connect request to tcp:www.googleapis.com:443
2024/05/16 22:05:41 [Info] [3569570120] app/dispatcher: taking detour [proxy] for [tcp:www.googleapis.com:443]
2024/05/16 22:05:41 [Info] [3569570120] transport/internet/tcp: dialing TCP to tcp:www.foobar.com:443
2024/05/16 22:05:41 [Debug] transport/internet: dialing to tcp:www.foobar.com:443
2024/05/16 22:05:41 tcp:127.0.0.1:50583 accepted tcp:www.googleapis.com:443 [socks-in -> proxy]
2024/05/16 22:05:41 [Info] [144733980] proxy: XtlsPadding 7 902 0
2024/05/16 22:05:41 [Info] [2607120133] app/proxyman/inbound: connection ends > proxy/socks: connection ends > context canceled
2024/05/16 22:05:43 [Info] [3569570120] proxy/vless/outbound: tunneling request to tcp:www.googleapis.com:443 via www.foobar.com:443
2024/05/16 22:05:43 [Info] [3569570120] proxy: XtlsFilterTls found tls client hello! 538
2024/05/16 22:05:43 [Info] [3569570120] proxy: XtlsPadding 538 434 0
2024/05/16 22:05:43 [Info] [3569570120] proxy: XtlsPadding 7 945 0
2024/05/16 22:05:45 [Info] [3233837674] proxy/socks: TCP Connect request to tcp:ssl.gstatic.com:443
2024/05/16 22:05:45 [Info] [3233837674] app/dispatcher: taking detour [proxy] for [tcp:ssl.gstatic.com:443]
2024/05/16 22:05:45 tcp:127.0.0.1:50585 accepted tcp:ssl.gstatic.com:443 [socks-in -> proxy]
2024/05/16 22:05:45 [Info] [3233837674] transport/internet/tcp: dialing TCP to tcp:www.foobar.com:443
2024/05/16 22:05:45 [Debug] transport/internet: dialing to tcp:www.foobar.com:443
2024/05/16 22:05:45 [Info] [3233837674] proxy/vless/outbound: tunneling request to tcp:ssl.gstatic.com:443 via www.foobar.com:443
2024/05/16 22:05:45 [Info] [3233837674] proxy: XtlsFilterTls found tls client hello! 567
2024/05/16 22:05:45 [Info] [3233837674] proxy: XtlsPadding 567 444 0
2024/05/16 22:05:45 [Info] [3233837674] proxy: XtlsPadding 7 933 0
2024/05/16 22:05:45 [Info] [627615421] proxy/socks: TCP Connect request to tcp:ssl.gstatic.com:443
2024/05/16 22:05:45 [Info] [627615421] app/dispatcher: taking detour [proxy] for [tcp:ssl.gstatic.com:443]
2024/05/16 22:05:45 [Info] [627615421] transport/internet/tcp: dialing TCP to tcp:www.foobar.com:443
2024/05/16 22:05:45 [Debug] transport/internet: dialing to tcp:www.foobar.com:443
2024/05/16 22:05:45 tcp:127.0.0.1:50587 accepted tcp:ssl.gstatic.com:443 [socks-in -> proxy]
2024/05/16 22:05:45 [Info] [433614612] proxy/socks: TCP Connect request to tcp:www.googleapis.com:443
2024/05/16 22:05:45 [Info] [433614612] app/dispatcher: taking detour [proxy] for [tcp:www.googleapis.com:443]
2024/05/16 22:05:45 [Info] [433614612] transport/internet/tcp: dialing TCP to tcp:www.foobar.com:443
2024/05/16 22:05:45 [Debug] transport/internet: dialing to tcp:www.foobar.com:443
2024/05/16 22:05:45 tcp:127.0.0.1:50589 accepted tcp:www.googleapis.com:443 [socks-in -> proxy]
2024/05/16 22:05:46 [Info] [433614612] proxy/vless/outbound: tunneling request to tcp:www.googleapis.com:443 via www.foobar.com:443
2024/05/16 22:05:46 [Info] [433614612] proxy: XtlsFilterTls found tls client hello! 602
2024/05/16 22:05:46 [Info] [433614612] proxy: XtlsPadding 602 459 0
2024/05/16 22:05:46 [Info] [433614612] proxy: XtlsPadding 7 1021 0
2024/05/16 22:05:46 [Info] [555232116] proxy/socks: TCP Connect request to tcp:www.googleapis.com:443
2024/05/16 22:05:46 [Info] [555232116] app/dispatcher: taking detour [proxy] for [tcp:www.googleapis.com:443]
2024/05/16 22:05:46 [Info] [555232116] transport/internet/tcp: dialing TCP to tcp:www.foobar.com:443
2024/05/16 22:05:46 [Debug] transport/internet: dialing to tcp:www.foobar.com:443
2024/05/16 22:05:46 tcp:127.0.0.1:50591 accepted tcp:www.googleapis.com:443 [socks-in -> proxy]
2024/05/16 22:05:47 [Info] [627615421] proxy/vless/outbound: tunneling request to tcp:ssl.gstatic.com:443 via www.foobar.com:443
2024/05/16 22:05:47 [Info] [627615421] proxy: XtlsFilterTls found tls client hello! 567
2024/05/16 22:05:47 [Info] [627615421] proxy: XtlsPadding 567 423 0
2024/05/16 22:05:47 [Info] [627615421] proxy: XtlsPadding 7 1300 0
2024/05/16 22:05:47 [Info] [555232116] proxy/vless/outbound: tunneling request to tcp:www.googleapis.com:443 via www.foobar.com:443
2024/05/16 22:05:47 [Info] [555232116] proxy: XtlsFilterTls found tls client hello! 538
2024/05/16 22:05:47 [Info] [555232116] proxy: XtlsPadding 538 821 0
2024/05/16 22:05:48 [Info] [555232116] proxy: XtlsPadding 7 1291 0
2024/05/16 22:05:48 [Info] [555232116] app/proxyman/outbound: failed to process outbound traffic > proxy/vless/outbound: connection ends > proxy/vless/outbound: failed to transfer response payload > read tcp 172.16.104.31:50592->172.67.139.124:443: read: connection reset by peer

Server log

[Fangliding]

Can it access http website?

[sky-chen]

yes. Both http and https works fine

[Fangliding]

I mean use this node to access http website(since it will get ssl err when accessing https)
btw pls provide the config, even you are flowing tutorial

[sky-chen]

can't access http sites, a download popup but can't finish the download due to network issue.

yep sure here's the client config (for the sake of completeness, full config provided)

// REFERENCE:
// https://github.com/XTLS/Xray-examples
// https://xtls.github.io/config/
// 常用的config文件，不论服务器端还是客户端，都有5个部分。外加小小白解读：

// ┌─ 1_log          日志设置 - 日志写什么，写哪里（出错时有据可查）

// ├─ 2_dns          DNS-设置 - DNS怎么查（防DNS污染、防偷窥、避免国内外站匹配到国外服务器等）

// ├─ 3_routing      分流设置 - 流量怎么分类处理（是否过滤广告、是否国内外分流）

// ├─ 4_inbounds     入站设置 - 什么流量可以流入Xray

// └─ 5_outbounds    出站设置 - 流出Xray的流量往哪里去
{

// 1_日志设置

// 注意，本例中我默认注释掉了日志文件，因为windows, macOS, Linux 需要写不同的路径，请自行配置

"log": {

// "access": "/home/local/xray_log/access.log",    // 访问记录

// "error": "/home/local/xray_log/error.log",    // 错误记录

"loglevel": "debug" // 内容从少到多: "none", "error", "warning", "info", "debug"

},
// 2_DNS设置

"dns": {

"servers": [

// 2.1 国外域名使用国外DNS查询

{

"address": "1.1.1.1",

"domains": ["geosite:geolocation-!cn"]

},

// 2.2 国内域名使用国内DNS查询，并期待返回国内的IP，若不是国内IP则舍弃，用下一个查询

{

"address": "223.5.5.5",

"domains": ["geosite:cn"],

"expectIPs": ["geoip:cn"]

},

// 2.3 作为2.2的备份，对国内网站进行二次查询

{

"address": "114.114.114.114",

"domains": ["geosite:cn"]

},

// 2.4 最后的备份，上面全部失败时，用本机DNS查询

"localhost"

]

},
// 3_分流设置

// 所谓分流，就是将符合否个条件的流量，用指定tag的出站协议去处理（对应配置的5.x内容）

"routing": {

"domainStrategy": "IPIfNonMatch",

"rules": [

// 3.1 广告域名屏蔽

{

"type": "field",

"domain": ["geosite:category-ads-all"],

"outboundTag": "block"

},

// 3.2 国内域名直连

{

"type": "field",

"domain": ["geosite:cn"],

"outboundTag": "direct"

},

// 3.3 国内IP直连

{

"type": "field",

"ip": ["geoip:cn", "geoip:private"],

"outboundTag": "direct"

},

// 3.4 国外域名代理

{

"type": "field",

"domain": ["geosite:geolocation-!cn"],

"outboundTag": "proxy"

},

// 3.5 默认规则

// 在Xray中，任何不符合上述路由规则的流量，都会默认使用【第一个outbound（5.1）】的设置，所以一定要把转发VPS的outbound放第一个

// 3.6 走国内"223.5.5.5"的DNS查询流量分流走direct出站

{

"type": "field",

"ip": ["223.5.5.5"],

"outboundTag": "direct"

}

]

},
// 4_入站设置

"inbounds": [

// 4.1 一般都默认使用socks5协议作本地转发

{

"tag": "socks-in",

"protocol": "socks",

"listen": "127.0.0.1", // 这个是通过socks5协议做本地转发的地址

"port": 1080, // 这个是通过socks5协议做本地转发的端口

"settings": {

"udp": true

}

}

],
// 5_出站设置

"outbounds": [

// 5.1 默认转发VPS

// 一定放在第一个，在routing 3.5 里面已经说明了，这等于是默认规则，所有不符合任何规则的流量都走这个

{

"tag": "proxy",

"protocol": "vless",

"settings": {

"vnext": [

{

"address": "www.foobar.com", // 替换成你的真实域名

"port": 443,

"users": [

{

"id": "36304791-f66f-xxx",

"flow": "xtls-rprx-vision",

"encryption": "none"

}

]

}

]

},

"streamSettings": {

"network": "tcp",

"security": "tls",

"tlsSettings": {

"serverName": "www.foobar.com", // 替换成你的真实域名

"allowInsecure": false, // 禁止不安全证书

"fingerprint": "chrome", // 通过 uTLS 库 模拟 Chrome / Firefox / Safari 或随机生成的指纹

"minVersion": "1.2"

}

}

},

// 5.2 用freedom协议直连出站，即当routing中指定'direct'流出时，调用这个协议做处理

{

"tag": "direct",

"protocol": "freedom"

},

// 5.3 用blackhole协议屏蔽流量，即当routing中指定'block'时，调用这个协议做处理

{

"tag": "block",

"protocol": "blackhole"

}

]

}

and server config

// REFERENCE:
// https://github.com/XTLS/Xray-examples
// https://xtls.github.io/config/
// 常用的 config 文件，不论服务器端还是客户端，都有 5 个部分。外加小小白解读：
// ┌─ 1*log 日志设置 - 日志写什么，写哪里（出错时有据可查）
// ├─ 2_dns DNS-设置 - DNS 怎么查（防 DNS 污染、防偷窥、避免国内外站匹配到国外服务器等）
// ├─ 3_routing 分流设置 - 流量怎么分类处理（是否过滤广告、是否国内外分流）
// ├─ 4_inbounds 入站设置 - 什么流量可以流入 Xray
// └─ 5_outbounds 出站设置 - 流出 Xray 的流量往哪里去
{
  // 1\_日志设置
  "log": {
    "loglevel": "debug", // 内容从少到多: "none", "error", "warning", "info", "debug"
    "access": "/home/my/xray_log/access.log", // 访问记录
    "error": "/home/my/xray_log/error.log" // 错误记录
  },
  // 2_DNS 设置
  "dns": {
    "servers": [
      "https+local://1.1.1.1/dns-query", // 首选 1.1.1.1 的 DoH 查询，牺牲速度但可防止 ISP 偷窥
      "localhost"
    ]
  },
  // 3*分流设置
  "routing": {
    "domainStrategy": "IPIfNonMatch",
    "rules": [
      // 3.1 防止服务器本地流转问题：如内网被攻击或滥用、错误的本地回环等
      {
        "type": "field",
        "ip": [
          "geoip:private" // 分流条件：geoip 文件内，名为"private"的规则（本地）
        ],
        "outboundTag": "block" // 分流策略：交给出站"block"处理（黑洞屏蔽）
      },
      {
        // 3.2 防止服务器直连国内
        "type": "field",
        "ip": ["geoip:cn"],
        "outboundTag": "block"
      },
      // 3.3 屏蔽广告
      {
        "type": "field",
        "domain": [
          "geosite:category-ads-all" // 分流条件：geosite 文件内，名为"category-ads-all"的规则（各种广告域名）
        ],
        "outboundTag": "block" // 分流策略：交给出站"block"处理（黑洞屏蔽）
      }
    ]
  },
  // 4*入站设置
  // 4.1 这里只写了一个最简单的 vless+xtls 的入站，因为这是 Xray 最强大的模式。如有其他需要，请根据模版自行添加。
  "inbounds": [
    {
      "port": 443,
      "protocol": "vless",
      "settings": {
        "clients": [
          {
            "id": "36304791-f66f-xxx", // 填写你的 UUID
            "flow": "xtls-rprx-vision",
            "level": 0,
            "email": "vpsadmin@yourdomain.com"
          }
        ],
        "decryption": "none",
        "fallbacks": [
          {
            "dest": 80 // 默认回落到防探测的代理
          }
        ]
      },
      "streamSettings": {
        "network": "tcp",
        "security": "tls",
        "tlsSettings": {
          "alpn": "http/1.1",
          "certificates": [
            {
              "certificateFile": "/home/my/xray_cert/xray.crt",
              "keyFile": "/home/my/xray_cert/xray.key"
            }
          ],
	  "minVersion": "1.2"
        }
      }
    }
  ],
  // 5*出站设置
  "outbounds": [
    // 5.1 第一个出站是默认规则，freedom 就是对外直连（vps 已经是外网，所以直连）
    {
      "tag": "direct",
      "protocol": "freedom"
    },
    // 5.2 屏蔽规则，blackhole 协议就是把流量导入到黑洞里（屏蔽）
    {
      "tag": "block",
      "protocol": "blackhole"
    }
  ]
}

[kingwilliam]

          "certificates": [
            {
              "certificateFile": "/home/my/xray_cert/xray.crt",
              "keyFile": "/home/my/xray_cert/xray.key"
            }
          ],

your xray.crt is self signed cert or real cert?

[sky-chen]

          "certificates": [
            {
              "certificateFile": "/home/my/xray_cert/xray.crt",
              "keyFile": "/home/my/xray_cert/xray.key"
            }
          ],

your xray.crt is self signed cert or real cert?

Cert from let's encrypt. It should be set up correctly cuz my site https is working, also I can see from the browser that my site cert is valid.

[Fangliding]

server log？

[sky-chen]

server log？

I restart xray, try tunnel (tried visiting both http, https sites). there is no log for tunnel part, only initial log for restart. here's error log, access log is always empty even though I set log level = "debug"

2024/05/18 08:36:20 [Debug] app/log: Logger started
2024/05/18 08:36:20 [Info] app/dns: DNS: created Local DOH client for https://1.1.1.1/dns-query
2024/05/18 08:36:20 [Info] app/dns: DNS: created localhost client
2024/05/18 08:36:20 [Debug] app/router: MphDomainMatcher is enabled for 719 domain rule(s)
2024/05/18 08:36:20 [Debug] app/proxyman/inbound: creating stream worker on 0.0.0.0:443
2024/05/18 08:36:20 [Info] transport/internet/tcp: listening TCP on 0.0.0.0:443
2024/05/18 08:36:20 [Warning] core: Xray 1.8.11 started
2024/05/18 08:36:24 [Info] [4069969794] proxy/vless/inbound: firstLen = 0
2024/05/18 08:36:24 [Info] [4069969794] proxy/vless/inbound: fallback starts > proxy/vless/inbound: fallback directly
2024/05/18 08:36:24 [Info] [4069969794] proxy/vless/inbound: realName = 
2024/05/18 08:36:24 [Info] [4069969794] proxy/vless/inbound: realAlpn = 
2024/05/18 08:36:25 [Info] [4069969794] app/proxyman/inbound: connection ends > proxy/vless/inbound: fallback ends > context canceled

Interestingly I check browser response and see http/0.9 200 ok, which lead me to a so page that include this

I've found that I get the same 'HTTP/0.9 200 OK' response if I try to connect to the SSL port (443) but specifying 'http' as the protocol.

I wonder that could shed light on the cause, perhaps client - server, or server - target_site is communicating with incorrect protocol?

[Fangliding]

"www.foobar.com"
Is this your site?

[sky-chen]

"www.foobar.com" Is this your site?

no. I made it up

[WordsWorthLess]

I think the 'alpn' object in the server config should be an array.

"alpn": "http/1.1",

I am surprised that xray service is up and running despite of the incorrect alpn setting.

[sky-chen]

solved. I use cloudflare to manage domain, which automatically setup proxy for my site, so tunnel traffic actually route to cloudflare proxy rather to my VPS. Just a head up for people with cloudflare

[Fangliding]

damn cloudflare problem

[yuhan6665]

Love it when capable user can resolve issues by themselves :)