Data Continuity Matters: Improving Sequence Modeling with Lipschitz Regularizer

This website provides the official implementation of the Lipschitz Regularizer from the following paper.

Data Continuity Matters: Improving Sequence Modeling with Lipschitz Regularizer
Eric Qu, Xufang Luo, Dongsheng Li
International Conference on Learning Representations, 2023. Notable top-25% (Spotlight).

Code Download



The aim of the Lipschitz Regularizer is to alter the continuity of the input sequence for sequence models during training. It is often applied after the embedding layer. The Lipschitz Regularizer can be calculated with the following function (works in PyTorch):

def LipReg(x0, transpose = True, normalize = True):
	x0: (B H L) if transposed else (B L H)

	the Lipschitz Regularizer, a scaler
	if not transpose:
		x0 = x0.transpose(-1, -2)
	if normalize:
		x = (x0 - x0.min()) / x0.max()
		x = x0
	return torch.square(x[..., 1:] - x[..., :-1]).mean(dim = -1).mean()

The normal way to use this regularizer is to calculate and return it in forward function in the model, and add it in the loss during training. Here is a pseudocode example:

class SomeModel(nn.Module):
	def __init__(...):

	def forward(self, x):
		h = self.embd(x) # calculate the embedding
		lip = LipReg(h)  # calculate the regularizer
		return output, lip


def train(...):
	for data, label in loader:
		output, lip = model(data)

		loss = lossfn(output, label) +/- lam * lip 
		# +: make input continuous; -: make input discrete


Example Usage

We provide an example usage with Informer in Code/Informer + LipReg/.


If you use this code, or otherwise found our work valuable, please cite:

	title={Data Continuity Matters: Improving Sequence Modeling with Lipschitz Regularizer},
	author={Eric Qu and Xufang Luo and Dongsheng Li},
	booktitle={International Conference on Learning Representations},
Last Updated: 4/5/2023, 6:36:10 PM