wget和urllib无法从python 3上的url加载文件

vatpfxk5  于 4个月前  发布在  Python
关注(0)|答案(3)|浏览(148)

我正在尝试下载一个txt文件，给出了一个url和端口。这在python 2上可以这样做：

Python 2.7.12 (default, Sep 26 2016, 09:46:23)  [GCC 4.2.1 Compatible
Apple LLVM 7.3.0 (clang-703.0.31)] on darwin Type "help", "copyright",
"credits" or "license" for more information.
>>> import urllib
>>> foo = urllib.urlopen("http://catnet-ip.icc.cat:8080/")
>>> foo.read() 
'SOURCETABLE 200 OK\r\nServer: NTRIP Trimble NTRIP Caster\r\nContent-Type: text/plain\r\nContent-Length: 2884\r\nDate:
02/Nov/2016:12:52:19 UTC\r\n\r\nSTR;VRS_RTK_2_3;Virtual RTK ver RTCM
2.3;RTCM 2.3;1(1),3(6),18(1),19(1),23(5),24(5);2;GPS;Catnet;ESP;41.3;2.09;1;1;Trimble
GPSNet;None;B;N;3900;;\r\nSTR;VRS_RTK_3_0;Virtual RTK ver RTCM
3.0;RTCM 3;1004(1),1005/1007(5),PBS(10);2;GPS;Catnet;ESP;41.3;2.09;1;1;Trimble
GPSNet;None;B;N;1100;;\r\nSTR;VRS_DGPS;Virtual DGPS ver RTCM 2.3;RTCM
2.3;1(1),3(6),22(6),23/24(5),16(59);0;GPS;Catnet;ESP;41.3;2.09;1;1;Trimble
GPSNet;None;B;N;640;;\r\n 
...

字符串
与wget类似：

Python 2.7.12 (default, Sep 26 2016, 09:46:23)  [GCC 4.2.1 Compatible
Apple LLVM 7.3.0 (clang-703.0.31)] on darwin Type "help", "copyright",
"credits" or "license" for more information.
>>> import wget
>>> foo = wget.download("http://catnet-ip.icc.cat:8080/", bar=None)
>>> foo
>>> ' (1).'
>>> exit()
$ less \ \(1\).
SOURCETABLE 200 OK\r\nServer: NTRIP Trimble NTRIP Caster\r\nContent-Type: text/plain\r\nContent-Length: 2884\r\nDate:
02/Nov/2016:12:52:19 UTC\r\n\r\nSTR;VRS_RTK_2_3;Virtual RTK ver RTCM
2.3;RTCM 2.3;1(1),3(6),18(1),19(1),23(5),24(5);2;GPS;Catnet;ESP;41.3;2.09;1;1;Trimble
GPSNet;None;B;N;3900;;\r\nSTR;VRS_RTK_3_0;Virtual RTK ver RTCM
3.0;RTCM 3;1004(1),1005/1007(5),PBS(10);2;GPS;Catnet;ESP;41.3;2.09;1;1;Trimble
GPSNet;None;B;N;1100;;\r\nSTR;VRS_DGPS;Virtual DGPS ver RTCM 2.3;RTCM
2.3;1(1),3(6),22(6),23/24(5),16(59);0;GPS;Catnet;ESP;41.3;2.09;1;1;Trimble
GPSNet;None;B;N;640;;\r\n 
...

型
但是在python 3上都失败了，错误是“http.client.BadStatusLine：SOURCETABLE 200 OK”。

Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:47:25)  [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request
>>> foo = urllib.request.urlopen("http://catnet-ip.icc.cat:8080/") 
Traceback (most recent call last):   
File "<stdin>", line 1, in <module>       
File /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)   
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 466, in open
    response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 484, in _open
    '_open', req)   File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1282, in http_open
    return self.do_open(http.client.HTTPConnection, req)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1257, in do_open
    r = h.getresponse()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1197, in getresponse
    response.begin()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 279, in _read_status
    raise BadStatusLine(line) 
http.client.BadStatusLine: SOURCETABLE 200 OK

型
以及：

Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:47:25)  [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information.
>>> import wget
>>> wget.download("http://catnet-ip.icc.cat:8080/")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/toni/Downloads/wget-2.0/wget.py", line 308, in download
    (tmpfile, headers) = urllib.urlretrieve(url, tmpfile, callback)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 188, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 466, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 484, in _open
    '_open', req)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1282, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1257, in do_open
    r = h.getresponse()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1197, in getresponse
    response.begin()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 279, in _read_status
    raise BadStatusLine(line)
http.client.BadStatusLine: SOURCETABLE 200 OK

型
从http协议的python文档中，我猜这是由于urllib和wget理解了我想作为http代码加载的文件的第一个位置的标签“SOURCETABLE”。这个标签总是出现在我想下载的文件中（ntrip casters），但我找不到解决这个问题的方法。

python-3.x

3条答案

按热度按时间
camsedfj

camsedfj1#

我在另一个NTRIP服务器上遇到了这个问题。根据RFC 2616，SOURCETABLE 200 OK不是有效的HTTP状态码。唉。我的解决办法是：curl，特别是pycurl。
举例来说：

import sys
import pycurl

def handle_write(buf):
    sys.stdout.write(buf.decode("iso-8859-1"))

host = "http://catnet-ip.icc.cat:8080/"
curl = pycurl.Curl()
curl.setopt(pycurl.URL, host)
curl.setopt(pycurl.TIMEOUT, 20)
curl.setopt(pycurl.CONNECTTIMEOUT, 3)
curl.setopt(pycurl.HEADERFUNCTION, sys.stdout.write)
curl.setopt(pycurl.WRITEFUNCTION, handle_write)

curl.perform()
curl.close()

字符串
结果：

SOURCETABLE 200 OK
Server: GNSS Spider 7.4.0.8125/1.0
Date: Wed, 18 Dec 2019 21:29:11 GMT Standard Time
Content-Type: text/plain
Content-Length: 2667

STR;VRS_RTK_3_0;VRS_RTK_3_0;RTCM 
...
ENDSOURCETABLE

型

赞(0）分享  回复(0）举报  4个月前
iovurdzv

iovurdzv2#

1.酒店Via Urllib 3

import urllib3
    http = urllib3.PoolManager()
    foo = http.request('GET',"http://example.com:2101/")
    print(foo.data)

字符串
1.通过urllib按照下一个方法编辑文件，然后尝试以下操作

import urllib.request
f = urllib.request.urlopen('http://splcare.in:2101')
print(f.read())

型
1.通过http模块您可以修改python http库，以额外的状态码，您需要在{python programm folder}/lib/http/client.py中编辑文件
查找代码行

if not version.startswith("HTTP/") :
    self._close_conn()
    raise BadStatusLine(line)

型
修改以

if not (version.startswith("HTTP/") or version.startswith("SOURCETABLE")):
    self._close_conn()
    raise BadStatusLine(line)

型
以下行的另一个修改

elif version.startswith("HTTP/1."):
        self.version = 11   # use HTTP/1.1 code for HTTP/1.x where x>=1
    else:
        raise UnknownProtocol(version)

型
修改为

elif version.startswith("HTTP/1."):
        self.version = 11   # use HTTP/1.1 code for HTTP/1.x where x>=1
    elif  version.startswith("SOURCETABLE"):
        self.version = 12   # Use NTRIP  
    else:
        raise UnknownProtocol(version)

型
现在你可以使用下面的代码在python控制台上获取源表，但是你需要编辑更多的文件来支持NTRIP。

import requests
url = "http://example.com:2101"
username = "xxxx"
password = "xxxx"
response = requests.get(url, auth=(username, password))
print(response.status_code)
print(response.content)

型

赞(0）分享  回复(0）举报  4个月前
wpcxdonn

wpcxdonn3#

另一种方法是使用标准库中的原始sockets：

with socket.create_connection(("catnet-ip.icc.cat", 2101)) as sock:
    req_payload = b"GET / HTTP/1.1\r\nHost: catnet-ip.icc.cat\r\n\r\n"
    sock.sendall(req_payload)

    resp_payload = sock.recv(4096)
    while data := sock.recv(4096):
        resp_payload += data

字符串
现在你可以自己解析resp_payload，或者让http.client.HTTPResponse像上面提到的here那样解析，但是需要做一些修改来克服SOURCETABLE状态行：

SOURCETABLE = b"SOURCETABLE"

http_payload = (
    resp_payload.replace(SOURCETABLE, b"HTTP/1.1".ljust(len(SOURCETABLE)), 1)
    if resp_payload.startswith(SOURCETABLE)
    else resp_payload
)

型

赞(0）分享  回复(0）举报  4个月前
我来回答

相关问题

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com