[Python知识库]Python_yield_实战应用_读取大文件

• 当我们在操作大文件(GB以上)时，最简单的是要把文件的一行一行全部取出来放到容器里面，这时会加载这个大文件到我们的内存空间中，但我们可能担心自己电脑内存不足。使用yield这个关键字便可以优雅的处理这个问题。
  关于yield的基础概念可以参阅
  https://blog.csdn.net/wyh1618/article/details/120371494?spm=1001.2014.3001.5501

def read_tab(fp: str, n: int) -> Iterable[List[str]]:
    i = 0
    lines = []  # a buffer to cache lines
    with open(fp, mode='rb') as f:
        # f = f.read()
        next(f)
        for line in f:
            i += 1
            lines.append(line.strip())  # append a line
            if i >= n:
                yield lines
                # reset buffer
                i = 0
                lines.clear()
    # remaining lines
    if i > 0:
        yield lines


lines_gen = read_tab(filename, 1000)  # 1000_yield
for lines in lines_gen:
    for each_l in lines:
        each_l_str = str(each_l, encoding="gbk")
        # print(each_l_str)
        each_l_list = list(each_l_str.split('\t'))

• 再就是注意在Linux机器下读文件时要注意编码问题，跟Windows下处理文件还是有点区别的。

def save_json_file(filename):
    lines_gen = read_tab(filename, 1000) 
    dict_all = dict()
    for lines in lines_gen:
        for each_l in lines:
            each_l_str = str(each_l, encoding="gbk")
            each_l_str = each_l_str.split('\t')
            dict_all[each_l_str[0]] = each_l_str[5]
    jsObj = json.dumps(dict_all, indent=4, ensure_ascii=False)
    fileObject = open('JsonFileNewTab.json', mode='w', encoding='utf-8')
    fileObject.write(jsObj)
    fileObject.close()

• Python处理Json小tips:

1. dumps()：将python中的 字典 转换为 字符串
2. loads(): 将 字符串 转换为 字典
3. dump(): 将数据写入json文件中
4. load():把文件打开，并把字符串变换为数据类型

def find_diffs_tab(file_name):
    old_files = {}
    with open('JsonFileOldTab.json', mode='r', encoding='utf-8') as load_old:
        old_files = json.load(load_old)

    new_files = {}
    with open('JsonFileNewTab.json', mode='r', encoding='utf-8') as load_new:
        new_files = json.load(load_new)

    all_have = old_files.keys() & old_files.keys()  # 共有的key
    not_all_have = old_files.keys() ^ new_files.keys()  # 不共有的key
    old_have_new_not = list(old_files.keys() - new_files.keys())  # 旧有新无
    new_have_old_not = list(new_files.keys() - old_files.keys())  # 旧无新有
    all_items = old_files.items() & new_files.items()  # 相同的键值对
    all_differ = set(old_files.items()) ^ set(new_files.items())  # 所有差异

    diff = old_files.keys() & new_files
    diff_values = [(k, old_files[k], new_files[k]) for k in diff if old_files[k] != new_files[k]]
    # 相同key，不同value

上述这个例子其实是处理两个大数据库文件某个字段的差异性，想起用Json去比较，比在数据库中两表直接join要快很多。

完整代码库整理完再更新。