RuntimeError:CUDA错误:CUBLAS_STATUS_EXECUTION_FAILED仅使用GPU调用`cublasSgemm(handle)`时

我正在研究具有一维信号的 CNN。它适用于 CPU 设备完全正常。但是，当我在 GPU 中训练模型时，发生了 CUDA 错误。我在调用os.environ['CUDA_LAUNCH_BLOCKING'] = "1"后设置命令。这样做时，发生了错误而不是错误。虽然 nvidia 文档怀疑硬件问题，但我可以用图像训练其他 CNN，没有任何错误。下面是我在训练模型中加载数据和设置数据的代码。RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILEDcublasCreate(handle)cublasSgemmcublasCreate

    idx = np.arange(len(dataset))  # dataset & label shuffle in once
    np.random.shuffle(idx)

    dataset = dataset[idx]
    sdnn = np.array(sdnn)[idx.astype(int)]        

    train_data, val_data = dataset[:int(0.8 * len(dataset))], dataset[int(0.8 * len(dataset)):]
    train_label, val_label = sdnn[:int(0.8 * len(sdnn))], sdnn[int(0.8 * len(sdnn)):]
    train_set = DataLoader(dataset=train_data, batch_size=opt.batch_size, num_workers=opt.workers)

    for i, data in enumerate(train_set, 0):  # data.shape = [batch_size, 3000(len(signal)), 1(channel)] tensor

        x = data.transpose(1, 2)
        label = torch.Tensor(train_label[i * opt.batch_size:i * opt.batch_size + opt.batch_size])
        x = x.to(device, non_blocking=True)
        label = label.to(device, non_blocking=True) # [batch size]
        label = label.view([len(label), 1])
        optim.zero_grad()

        # Feature of signal extract
        y_predict = model(x) # [batch size, fc3 output] # Error occurred HERE
        loss = mse(y_predict, label)

以下是此代码的错误消息。

File C:/Users/Me/Desktop/Me/Study/Project/Analysis/Regression/main.py", line 217, in Processing
    y_predict = model(x) # [batch size, fc3 output]
  File "C:Anacondaenvstorchlibsite-packagestorchnnmodulesmodule.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:UsersMEDesktopMEStudyProjectAnalysisRegressioncnn.py", line 104, in forward
    x = self.fc1(x)
  File "C:Anacondaenvstorchlibsite-packagestorchnnmodulesmodule.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:Anacondaenvstorchlibsite-packagestorchnnmoduleslinear.py", line 91, in forward
    return F.linear(input, self.weight, self.bias)
  File "C:Anacondaenvstorchlibsite-packagestorchnnfunctional.py", line 1674, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

数周以来，我一直试图解决此错误，但找不到解决方案。如果您在这里看到任何错误，请告诉我。

回答