0
0

微调大型语言模型示例：使用T5将自然语言转换成SQL语句

将自然语言转换为SQL语句已经不再遥不可及。NLP的进步使得我们不仅可以使用LLM(大型语言模型)，还可以通过微调教授他们新的技能，这也被称为迁移学习。可以使用一个预先训练的模型作为起点，然后使用较小的标记数据集从而获得比单独使用数据训练更好的性能。

在本文中，我们将使用谷歌的文本到文本生成模型T5和我们的自定义数据进行迁移学习，这样它就可以将基本问题转换为SQL查询。我们将在T5中添加一个名为:将英语翻译为SQL的新任务，它可以转换以下示例查询:

  1. Cars built after 2020 and manufactured in Italy

将输出一下SQL语句

  1. SELECT name FROM cars WHERE location = 'Italy' AND date > 2020

创建训练数据

与翻译数据集不同，我们可以在模板的帮助下以编程方式自动构建训练的数据集，下面是整理出来的一些模板：

  1. templates = [
  2.               ["[prop1] of [nns]","SELECT [prop1] FROM [nns]"],
  3.               ["[agg] [prop1] for each [breakdown]","SELECT [agg]([prop1]) , [breakdown] FROM [prop1] GROUP BY [breakdown]"],
  4.               ["[prop1] of [nns] by [breakdown]","SELECT [prop1] , [breakdown] FROM [nns] GROUP BY [breakdown]"],
  5.               ["[prop1] of [nns] in [location] by [breakdown]","SELECT [prop1] , [breakdown] FROM [nns] WHERE location = '[location]' GROUP BY [breakdown]"],
  6.               ["[nns] having [prop1] between [number1] and [number2]","SELECT name FROM [nns] WHERE [prop1] > [number1] and [prop1] < [number2]"],
  7.               ["[prop] by [breakdown]","SELECT name , [breakdown] FROM [prop] GROUP BY [breakdown]"],
  8.               ["[agg] of [prop1] of [nn]","SELECT [agg]([prop1]) FROM [nn]"],
  9.               ["[prop1] of [nns] before [year]","SELECT [prop1] FROM [nns] WHERE date < [year]"],
  10.               ["[prop1] of [nns] after [year] in [location]","SELECT [prop1] FROM [nns] WHERE date > [year] AND location='[location]'"],
  11.               ["[nns] [verb] after [year] in [location]","SELECT name FROM [nns] WHERE location = '[location]' AND date > [year]"],
  12.               ["[nns] having [prop1] between [number1] and [number2] by [breakdown]","SELECT name , [breakdown] FROM [nns] WHERE [prop1] < [number1] AND [prop1] > [number2] GROUP BY [breakdown]"],
  13.               ["[nns] with a [prop1] of maximum [number1] by their [breakdown]","SELECT name , [breakdown] FROM [nns] WHERE [prop1] <= [number1] GROUP BY [breakdown]"],
  14.               ["[prop1] and [prop2] of [nns] since [year]","SELECT [prop1] , [prop2] FROM [nns] WHERE date > [year]"],
  15.               ["[nns] which have both [prop1] and [prop2]","SELECT name FROM [nns] WHERE [prop1] IS true AND [prop2] IS true"],
  16.               ["Top [number1] [nns] by [prop1]","SELECT name FROM [nns] ORDER BY [prop1] DESC LIMIT [number1]"]
  17. ]
  18. template = random.choice(templates)
  19. print("Sample Query Template  :", template[0])
  20. print("SQL Translation        :", template[1])

构建函数使用这些模板并生成我们的数据集

  1. objects = ["countries","wines","wineries","tasters", "provinces","grapes","cities","bottles","deliveries"]
  2. object_single = ["country","wine","winery","taster", "province","grape","city","bottle", "delivery"]
  3. properties = ["points","price","taste","title","texture","age","duration","acidity","flavor","level"]
  4. aggs = [["average","avg"], ["total","sum"],["count","count"], ["minimum","min"], ["maximum","max"]]
  5. breakdowns = ["quality","price","province","country","point", "variety","flavor","age"]
  6. locations = ["Italy","US","Portugal","Spain","Chile","Turkey","Canada"]
  7. verbs = ["produced","bottled"]
  9. regex = r"\[([a-z0-9]*)\]"
  10. number_of_samples = 2500
  12. @dataset("english_sql_translations")
  13. def build_dataset():
  14.     rows = []
  15.     for index in range(0,number_of_samples):
  16.         template = random.choice(templates)
  17.         nl = template[0]
  18.         sql = template[1]
  20.         matches = re.finditer(regex, nl, re.MULTILINE)
  22.         for matchNum, match in enumerate(matches, start=1):
  23.             key = match.group()
  24.             prop = None
  25.             prop_sql = None
  26.             if key.startswith("[prop"):
  27.                 prop = random.choice(properties)
  28.                 prop_sql = prop.replace(" ","_").lower()
  29.             if key in ["[nns]"]:
  30.                 prop = random.choice(objects)
  31.                 prop_sql = prop
  32.             if key in ["[nn]"]:
  33.                 prop = random.choice(object_single)
  34.                 prop_sql = prop.replace(" ","_").lower()
  35.             if key == "[breakdown]":
  36.                 prop = random.choice(breakdowns)
  37.                 prop_sql = prop.replace(" ","_").lower()
  38.             if key == "[verb]":
  39.                 prop = random.choice(verbs)
  40.                 prop_sql = prop.replace(" ","_").lower()
  41.             if key == "[agg]":
  42.                 aggregation = random.choice(aggs)
  43.                 prop = aggregation[0]
  44.                 prop_sql = aggregation[1]
  45.             if key == "[location]":
  46.                 prop = random.choice(locations)
  47.                 prop_sql = prop
  48.             if key.startswith("[number"):
  49.                 prop = str(random.randint(1,1000))
  50.                 prop_sql = prop
  51.             if key.startswith("[year"):
  52.                 prop = str(random.randint(1950,2022))
  53.                 prop_sql = prop
  54.             
  56.             if prop is not None:
  57.                 nl = nl.replace(key,prop)
  58.                 sql = sql.replace(key,prop_sql)
  59.         
  60.         prefix = random.randint(1,20)
  61.         if prefix == 1:
  62.             nl = "Show me "+nl
  63.         elif prefix == 2:
  64.             nl = "List "+nl
  65.         elif prefix == 3:
  66.             nl = "List of "+nl
  67.         elif prefix == 4:
  68.             nl = "Find "+nl
  69.         rows.append([nl,sql])
  71.     df = pd.DataFrame(rows, columns=["query", "sql"])
  72.     return df

这里使用了@Dataset装饰器。现在，我们可以将此函数通过以下方式轻松分层：

  1. layer.run([build_dataset])

运行完成后，可以开始构建自定义数据集加载程序

创建数据加载程序

我们的数据集还需要使用Pytorch的Dataset实现，才能够使用Dataloader进行加载

  1. from torch.utils.data import Dataset
  2. class EnglishToSQLDataSet(Dataset):
  4.   def __init__(self, dataframe, tokenizer, source_len, target_len, source_text, target_text):
  5.     self.tokenizer = tokenizer
  6.     self.data = dataframe
  7.     self.source_len = source_len
  8.     self.target_len = target_len
  9.     self.target_text = self.data[target_text]
  10.     self.source_text = self.data[source_text]
  12.     self.data["query"] = "translate English to SQL: "+self.data["query"]
  13.     self.data["sql"] = "<pad>" + self.data["sql"] + "</s>"
  15.   def __len__(self):
  16.     return len(self.target_text)
  18.   def __getitem__(self, index):
  19.     source_text = str(self.source_text[index])
  20.     target_text = str(self.target_text[index])
  22.     source_text = ' '.join(source_text.split())
  23.     target_text = ' '.join(target_text.split())
  25.     source = self.tokenizer.batch_encode_plus([source_text], max_length= self.source_len, pad_to_max_length=True, truncation=True, padding="max_length", return_tensors='pt')
  26.     target = self.tokenizer.batch_encode_plus([target_text], max_length= self.target_len, pad_to_max_length=True, truncation=True, padding="max_length", return_tensors='pt')
  28.     source_ids = source['input_ids'].squeeze()
  29.     source_mask = source['attention_mask'].squeeze()
  30.     target_ids = target['input_ids'].squeeze()
  31.     target_mask = target['attention_mask'].squeeze()
  33.     return {
  34.         'source_ids': source_ids.to(dtype=torch.long),
  35.         'source_mask': source_mask.to(dtype=torch.long),
  36.         'target_ids': target_ids.to(dtype=torch.long),
  37.         'target_ids_y': target_ids.to(dtype=torch.long)
  38.     }

微调 T5

数据集已准备就完毕。现在可以开发微调逻辑。用@model对功能进行装饰，然后将其传递给层。

  1. ef train(epoch, tokenizer, model, device, loader, optimizer):
  2.     import torch
  4.     model.train()
  5.     for _,data in enumerate(loader, 0):
  6.         y = data['target_ids'].to(device, dtype = torch.long)
  7.         y_ids = y[:, :-1].contiguous()
  8.         lm_labels = y[:, 1:].clone().detach()
  9.         lm_labels[y[:, 1:] == tokenizer.pad_token_id] = -100
  10.         ids = data['source_ids'].to(device, dtype = torch.long)
  11.         mask = data['source_mask'].to(device, dtype = torch.long)
  13.         outputs = model(input_ids = ids, attention_mask = mask, decoder_input_ids=y_ids, labels=lm_labels)
  14.         loss = outputs[0]
  16.         step = (epoch * len(loader)) + _
  17.         layer.log({"loss": float(loss)}, step)
  19.         optimizer.zero_grad()
  20.         loss.backward()
  21.         optimizer.step()

在这里，我们使用三个单独装饰器：

@model：告诉图层使用此函数训练ML模型

@Fabric：告诉层训练模型所需的计算资源（CPU，GPU等）。T5是一个大型型号，我们需要GPU对其进行微调。

@pip_requirements：微调我们的模型所需的Python软件包。

  1. @model("t5-tokenizer")
  2. @fabric("f-medium")
  3. @pip_requirements(packages=["torch","transformers","sentencepiece"])
  4. def build_tokenizer():
  5.     from transformers import T5Tokenizer
  6.     # Load tokenizer from Hugging face
  7.     tokenizer = T5Tokenizer.from_pretrained("t5-small")
  8.     return tokenizer
  10. @model("t5-english-to-sql")
  11. @fabric("f-gpu-small")
  12. @pip_requirements(packages=["torch","transformers","sentencepiece"])
  13. def build_model():
  14.     from torch.utils.data import Dataset, DataLoader, RandomSampler, SequentialSampler
  15.     from transformers import T5Tokenizer, T5ForConditionalGeneration
  16.     import torch.nn.functional as F
  17.     from torch import cuda
  18.     import torch
  19.     
  20.     parameters={
  21.         "BATCH_SIZE":8,          
  22.         "EPOCHS":3,              
  23.         "LEARNING_RATE":2e-05,          
  24.         "MAX_SOURCE_TEXT_LENGTH":75,   
  25.         "MAX_TARGET_TEXT_LENGTH":75,
  26.         "SEED": 42
  27.     }
  29.     # Log parameters to Layer
  30.     layer.log(parameters)
  31.     
  32.     # Set seeds for reproducibility
  33.     torch.manual_seed(parameters["SEED"])
  34.     np.random.seed(parameters["SEED"])
  35.     torch.backends.cudnn.deterministic = True
  37.     # Load tokenizer from Layer
  38.     tokenizer = layer.get_model("t5-tokenizer").get_train()
  40.     # Load pretrained model from Hugging face
  41.     model = T5ForConditionalGeneration.from_pretrained("t5-small")
  42.     device = 'cuda' if cuda.is_available() else 'cpu'
  43.     model.to(device)
  45.     dataframe = layer.get_dataset("english_sql_translations").to_pandas()
  46.     source_text = "query"
  47.     target_text = "sql"
  49.     dataframe = dataframe[[source_text,target_text]]
  51.     train_dataset = dataframe.sample(frac=0.8,random_state = parameters["SEED"])
  52.     train_dataset = train_dataset.reset_index(drop=True)
  54.     layer.log({"FULL Dataset": str(dataframe.shape),
  55.              "TRAIN Dataset": str(train_dataset.shape)
  56.              })
  58.     training_set = EnglishToSQLDataSet(train_dataset, tokenizer, parameters["MAX_SOURCE_TEXT_LENGTH"], parameters["MAX_TARGET_TEXT_LENGTH"], source_text, target_text)
  60.     dataloader_paramaters = {
  61.       'batch_size': parameters["BATCH_SIZE"],
  62.       'shuffle': True,
  63.       'num_workers': 0
  64.       }
  66.     training_loader = DataLoader(training_set, **dataloader_paramaters)
  67.     optimizer = torch.optim.Adam(params =  model.parameters(), lr=parameters["LEARNING_RATE"])
  69.     for epoch in range(parameters["EPOCHS"]):
  70.         train(epoch, tokenizer, model, device, training_loader, optimizer)
  72.     return model

现在可以将代码传递到远程GPU实例上训练我们的模型。

  1. layer.run([build_tokenizer, build_model], debug=True)

训练完成后，我们可以在UI层中找到我们的模型和指标。这是我们的损失曲线：

使用Gradio构建演示

Gradio是演示机器学习模型的最快方法，任何人在任何地方都可以使用它！我们将与Gradio建立互动演示，以为人们提供UI想要尝试我们的模型。

让我们开始编码。创建一个名为app.py的python文件，并列出以下代码：

  1. import gradio as gr
  2. import layer
  4. model = layer.get_model('layer/t5-fine-tuning-with-layer/models/t5-english-to-sql').get_train()
  5. tokenizer = layer.get_model('layer/t5-fine-tuning-with-layer/models/t5-tokenizer').get_train()
  7. def greet(query):
  8.     input_ids = tokenizer.encode(f"translate English to SQL: {query}", return_tensors="pt")
  9.     outputs = model.generate(input_ids, max_length=1024)
  10.     sql = tokenizer.decode(outputs[0], skip_special_tokens=True)
  11.     return sql
  13. iface = gr.Interface(fn=greet, inputs="text", outputs="text", examples=[
  14.     "Show me the average price of wines in Italy by provinces",
  15.     "Cars built after 2020 and manufactured in Italy",
  16.     "Top 10 cities by their population"
  17. ])
  18. iface.launch()

在上面的代码中,使用Gradio创建一个简单的UI：一个用于查询输入的输入TextField和一个输出TextField以显示预测的SQL查询

我们将需要一些额外的库，所以需要创建一个具有以下内容的sumploy.txt文件：

  1. layer==0.9.350435
  2. torch==1.11.0
  3. sentencepiece==0.1.96

这样就可以发布我们的Gradio应用程序：

  1. 去Hugging Face创造一个space
  2. 输入Gradio的Key作为Space SDK

现在，克隆Hugging Face的space到本地目录中：

  1. $ git clone [YOUR_HUGGINGFACE_SPACE_URL]

将requirements.txt 和app.py文件放入克隆目录中，并在终端中运行以下命令：

  1. $ git add app.py
  2. $ git add requirements.txt
  3. $ git commit -m "Add application files"
  4. $ git push

现在前往Hugging Face的space，部署应用程序后，就可以看到如下的界面

最后

本文展示了如何微调大型语言模型来教他们新技能。我们可以设计自己的任务，并进行微调T5供自己使用。

本文的项目demo和完整代码在这里：

https://huggingface.co/spaces/mecevit/english-to-sql

https://app.layer.ai/layer/t5-fine-tuning-with-layer

作者：Mehmet Ecevit

标签： 自然语言处理 神经网络 深度学习
登录后发布评论

“微调大型语言模型示例：使用T5将自然语言转换成SQL语句”的评论:

还没有评论
关于作者
...
Deephub
更多文章请关注公众号：Deephub-IMBA
相关阅读

从零实现基于扩散模型的文本到视频生成系统：技术详解与Pytorch代码实现

18个常用的强化学习算法整理：从基础方法到高级模型的理论技术与代码实现

PyTorch CUDA内存管理优化：深度理解GPU资源分配与缓存机制

LangChain RAG入门教程：构建基于私有文档的智能问答助手

基于LlamaIndex实现CodeAct Agent：代码执行工作流的技术架构与原理

频率派与贝叶斯统计在营销组合建模中的应用比较：隐私优先时代的方法选择

基于Transformer架构的时间序列数据去噪技术研究