Introdução

Ideia Geral

Utilizando os dados de câncer da FOSP, somente do tipo colorretal, serão utilizados quatro modelos de machine learning diferentes, com o intuito de testar diferentes tipos de algoritmo na classificação do óbito por câncer.

O label é 0 se o paciente está vivo e 1 se ele morreu devido ao câncer.

Modelos de ML

Foram escolhidos os modelos Naive Bayes, que utiliza o Teorema de Bayes para realizar as previsões, Random Forest, XGBoost e LightGBM, que utilizam os conceitos de árvores de decisão, além de bagging e boosting. Além disso, será testado um modelo de votação com os melhores classificadores obtidos, visando obter um algoritmo ainda mais acertivo.

Validação dos modelos

Para validar os modelos treinados foi utilizada primeiramente a matriz de confusão, sendo possível avaliar os acertos em ambas as classes. Para entender de houve overfitting nos modelos, foi utilizada a curva ROC para os conjuntos de treino e teste, comparando a métrica AUC entre ambos os conjuntos.

Por fim, os modelos Random Forest, XGBoost e LightGBM oferecem a possibilidade de saber quais foram as features mais importantes, ou seja, que mais influenciam na previsão das classes. Assim, foram mostradas duas maneiras diferentes de analisar a importância das variáveis de entrada, uma usando a própria função dos modelos e outra usando a biblioteca SHAP, que mostra a influência das features em ambas as classes.

# Leitura dos dados
df = read_csv('/content/drive/MyDrive/Trabalho/Cancer/Datasets/colorretal_labels.csv')
df.head(3)

(31916, 37)

	ESCOLARI	IDADE	SEXO	IBGE	CATEATEND	DIAGPREV	EC	ECGRUP	TRATHOSP	NENHUM	...	IBGEATEN	ULTICONS	ULTIDIAG	ULTITRAT	obito_geral	obito_cancer	vivo_ano1	vivo_ano3	vivo_ano5	ESCOLARI_2
0	4	19	2	3538709	9	2	IV	IV	I	0	...	3538709	4985	4985	4951	0	0	1	1	1	4.0
1	9	19	1	3537107	2	2	IIIA	III	I	0	...	3509502	2680	2744	2674	1	1	1	1	1	4.0
2	4	19	1	3516200	9	2	IIB	II	F	0	...	3516200	4725	4734	4719	0	0	1	1	1	4.0

3 rows × 37 columns

# Valores faltantes
df.isna().sum().sort_values(ascending=False).head(6)

ESCOLARI     0
CONSDIAG     0
DIAGTRAT     0
ANODIAG      0
FAIXAETAR    0
DRS          0
dtype: int64

# Correlação com a saída
corr_matrix = df.corr()
abs(corr_matrix['obito_cancer']).sort_values(ascending = False).head(20)

The default value of numeric_only in DataFrame.corr is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning.

obito_cancer    1.000000
obito_geral     0.788496
ULTINFO         0.472244
ULTIDIAG        0.396297
ULTITRAT        0.394160
ULTICONS        0.392998
vivo_ano3       0.384454
vivo_ano5       0.365039
vivo_ano1       0.279604
RECNENHUM       0.231324
ANODIAG         0.175160
CATEATEND       0.158817
CIRURGIA        0.155553
QUIMIO          0.086075
RRAS            0.065750
ESCOLARI_2      0.056375
RADIO           0.051821
IBGEATEN        0.049655
DIAGPREV        0.047864
OUTROS          0.043201
Name: obito_cancer, dtype: float64

# Quantidade de pacientes em cada classe da saída
df.obito_cancer.value_counts()

0    18780
1    13136
Name: obito_cancer, dtype: int64

Análise - Óbito por câncer

Pré-processamento

Como o dataset já foi limpo anteriormente, aqui na etapa de pré-processamento serão realizadas a divisão dos dados em treino e teste, a codificação das colunas textuais para colunas numéricas e a normalização dos dados. Com isso, temos os dados prontos para o treinamento dos modelos de machine learning e consequentemente sua validação.

Neste primeiro momento, serão definidas as colunas que não serão utilizadas como features, assim, foi escolhido manter a coluna IDADE, então a coluna FAIXAETAR será retirada. O mesmo ocorre com a coluna EC, retirando a coluna ECGRUP. Por fim, as outras colunas contidas na list_drop são possíveis saídas para os modelos, mas estamos interessados somente no óbito por câncer, por isso só ela será mantida como label e as outras serão retiradas.

list_drop = ['FAIXAETAR', 'ULTICONS', 'ULTIDIAG', 'ULTITRAT', 'vivo_ano1',
             'vivo_ano3', 'vivo_ano5', 'ULTINFO', 'obito_geral', 'ECGRUP', 'ESCOLARI']

lb = 'obito_cancer'

Uma função foi criada para realizar o pré-processamento inteiro, chamada preprocessing, internamente ela utiliza outras funções criadas que são: get_train_test (divide os dados em treino e teste), train_preprocessing (prepara os dados de treino) e test_preprocessing (prepara os dados de teste).

Mais detalhes em funções.

X_train, X_test, y_train, y_test, feat_cols = preprocessing(df, list_drop, lb,
                                                            random_state=seed,
                                                            balance_data=False,
                                                            encoder_type='LabelEncoder',
                                                            norm_name='StandardScaler')

X_train = (23937, 25), X_test = (7979, 25)
y_train = (23937,), y_test = (7979,)

feat_cols

Index(['IDADE', 'SEXO', 'IBGE', 'CATEATEND', 'DIAGPREV', 'EC', 'TRATHOSP',
       'NENHUM', 'CIRURGIA', 'RADIO', 'QUIMIO', 'HORMONIO', 'TMO', 'IMUNO',
       'OUTROS', 'NENHUMANT', 'CONSDIAG', 'TRATCONS', 'DIAGTRAT', 'ANODIAG',
       'DRS', 'RRAS', 'RECNENHUM', 'IBGEATEN', 'ESCOLARI_2'],
      dtype='object')

y_train.value_counts(normalize=True)

0    0.58842
1    0.41158
Name: obito_cancer, dtype: float64

y_test.value_counts(normalize=True)

0    0.58842
1    0.41158
Name: obito_cancer, dtype: float64

Treinamento e validação dos modelos de machine learning

Depois das etapas de preparação, os dados estão prontos para serem utilizados nos modelos escolhidos.

Naive Bayes

# Criação e treinamento do modelo Naive Bayes
nb = GaussianNB()
nb.fit(X_train, y_train)

GaussianNB()

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# Matriz de confusão
plot_confusion_matrix(nb, X_test, y_test)

              precision    recall  f1-score   support

           0      0.702     0.013     0.025      4695
           1      0.413     0.992     0.583      3284

    accuracy                          0.416      7979
   macro avg      0.558     0.502     0.304      7979
weighted avg      0.583     0.416     0.255      7979

Claramente percebe-se que o modelo previu quase todos os dados como sendo da classe 1, portanto não teve um aprendizado satisfatório.

Na matriz de confusão, buscamos uma diagonal principal equilibrada e com a maior acertividade possível.

# Curva ROC
plot_roc_curve(nb, X_train, X_test, y_train, y_test)

Pelas curvas ROC, pode-se dizer que não há overfitting, mas o modelo é ruim para a previsão da classe 0, portanto qualquer análise além dessa não possui tanta relevância.

Random Forest

O modelo Random Forest é mais complexo em relação ao Naive Bayes, assim alguns hiperparâmetros serão definidos para obter um modelo base e depois será realizada a busca dos melhores parâmetros utilizando o Optuna.

Os parâmetros definidos para este primeiro modelo serão:

• random_state: para repetibilidade do treinamento do modelo. Será utilizado na busca pelos hiperparâmetros também, sempre como mesmo valor definido na variável seed.

• max_depth: será definido como 8, pois o padrão do modelo é não ter profundidade máxima para as árvores, o que dificulta e faz o treinamento ser muito longo, além da maior chance de overfitting.

• class_weight: usado para definir os pesos de cada classe no treinamento do modelo, muito útil quando temos classes desbalanceadas no conjunto de dados, como neste caso.

# Criação e treinamento do modelo Random Forest
rf = RandomForestClassifier(random_state=seed,
                            max_depth=8,
                            class_weight={0:1, 1:1.77},
                            criterion='entropy')

rf.fit(X_train, y_train)

RandomForestClassifier(class_weight={0: 1, 1: 1.77}, criterion='entropy',
                       max_depth=8, random_state=10)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# Matriz de confusão
plot_confusion_matrix(rf, X_test, y_test)

              precision    recall  f1-score   support

           0      0.817     0.757     0.786      4695
           1      0.686     0.757     0.720      3284

    accuracy                          0.757      7979
   macro avg      0.751     0.757     0.753      7979
weighted avg      0.763     0.757     0.759      7979

A matriz obtida para o modelo Random Forest apresentou diagonal equilibrada em ambas as classes, com 76% de acurácia.

# Pedaço de uma das árvores do modelo Random Forest
show_tree(rf, feat_cols, 2)

# Curva ROC
plot_roc_curve(rf, X_train, X_test, y_train, y_test)

Como a métrica AUC possui valores próximos para o conjunto de treino e de teste, 0,864 e 0,837 respectivamente, pode-se dizer que há apenas um pouco de overfitting, não sendo algo de grande preocupação.

# Importância das features
plot_feat_importances(rf, feat_cols)

As features mais importantes nesta visualização são EC, com uma grande vantagem, RECNENHUM, ANODIAG e CATEATEND.

# Importância das features pelos valores SHAP
plot_shap_values(rf, X_train, feat_cols)

A coluna EC foi a mais importante aqui também, com isso, os valores mais altos desta variável, mostrados em rosa, influenciaram mais o modelo na previsão da classe 1 (óbito por câncer). Já os valores mais baixos desta coluna, em azul, tem mais peso para previsão ser da classe 0. Este comportamento faz sentido, pois quanto mais alto o estágio, maior é a extensão do câncer, assim menor a chance de sobrevivência.

O raciocínio para analisar as outras colunas é o mesmo utilizado para o estadiamento clínico.

Optuna

Para fazer a busca pelos melhores hiperparâmetros, será utilizado a biblioteca Optuna, definindo o intervalo para os parâmetros do modelo a serem buscados.

# Folds com a mesma proporção das classes
skf = StratifiedKFold(10, shuffle=True, random_state=seed)

# Função com o modelos e seus parâmetros, que terá sua métrica maximizada
def objective(trial):

    n_estimators = trial.suggest_int('n_estimators', 50, 250)
    max_depth = trial.suggest_int('max_depth', 3, 18)
    min_samples_split = trial.suggest_int('min_samples_split', 2, 10)
    min_samples_leaf = trial.suggest_int('min_samples_leaf', 1, 7)
    max_samples = trial.suggest_float('max_samples', 0.7, 1.0, step=0.1)
    optimizer = trial.suggest_categorical('criterion', ['gini', 'entropy'])

    cls = RandomForestClassifier(n_estimators=n_estimators,
                                 max_depth=max_depth,
                                 min_samples_split=min_samples_split,
                                 min_samples_leaf=min_samples_leaf,
                                 max_samples=max_samples,
                                 criterion=optimizer,
                                 random_state=seed)

    return cross_val_score(cls, X_train, y_train,
                           cv=skf, scoring='balanced_accuracy').mean()

# Criação do estudo e procura pelos hiperparâmetros
studyRF = optuna.create_study(direction='maximize', sampler=RandomSampler(seed))
studyRF.optimize(objective, n_trials=100)

# Melhor tentativa
studyRF.best_trial

FrozenTrial(number=92, state=TrialState.COMPLETE, values=[0.759440917196401], datetime_start=datetime.datetime(2023, 4, 12, 0, 56, 52, 730637), datetime_complete=datetime.datetime(2023, 4, 12, 0, 58, 1, 911801), params={'n_estimators': 201, 'max_depth': 17, 'min_samples_split': 2, 'min_samples_leaf': 3, 'max_samples': 1.0, 'criterion': 'gini'}, user_attrs={}, system_attrs={}, intermediate_values={}, distributions={'n_estimators': IntDistribution(high=250, log=False, low=50, step=1), 'max_depth': IntDistribution(high=18, log=False, low=3, step=1), 'min_samples_split': IntDistribution(high=10, log=False, low=2, step=1), 'min_samples_leaf': IntDistribution(high=7, log=False, low=1, step=1), 'max_samples': FloatDistribution(high=1.0, log=False, low=0.7, step=0.1), 'criterion': CategoricalDistribution(choices=('gini', 'entropy'))}, trial_id=92, value=None)

# Melhores parâmetros
studyRF.best_params

{'n_estimators': 201,
 'max_depth': 17,
 'min_samples_split': 2,
 'min_samples_leaf': 3,
 'max_samples': 1.0,
 'criterion': 'gini'}

plot_optimization_history(studyRF).show()

# Modelo com os melhores parâmetros
params = studyRF.best_params
params['random_state'] = seed
params['class_weight'] = {0: 1, 1: 2.3}

rf_optuna = RandomForestClassifier()
rf_optuna.set_params(**params)

rf_optuna.fit(X_train, y_train)

RandomForestClassifier(class_weight={0: 1, 1: 2.3}, max_depth=17,
                       max_samples=1.0, min_samples_leaf=3, n_estimators=201,
                       random_state=10)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# Matriz de confusão do modelo Random Forest otimizado
plot_confusion_matrix(rf_optuna, X_test, y_test)

              precision    recall  f1-score   support

           0      0.825     0.768     0.796      4695
           1      0.699     0.768     0.731      3284

    accuracy                          0.768      7979
   macro avg      0.762     0.768     0.764      7979
weighted avg      0.773     0.768     0.769      7979

Há uma melhora de acurácia em relação ao primeiro modelo testado, de 76% para 77%.

# Curva ROC do modelo otimizado
plot_roc_curve(rf_optuna, X_train, X_test, y_train, y_test)

A curva ROC mostra que o modelo possui overfitting, pois para o conjunto de treino temos AUC = 0,974 e para o teste AUC = 0,844, essa diferença caracteriza o problema.

# Importância das features pelos valores SHAP, para o modelo com os melhores hiperparâmetros
plot_shap_values(rf_optuna, X_train, feat_cols)

No data for colormapping provided via 'c'. Parameters 'vmin', 'vmax' will be ignored

XGBoost

O modelo XGBoost também terá alguns hiperparâmetros definidos para obter um modelo base e depois será realizada a busca dos melhores parâmetros utilizando o Optuna.

Os parâmetros definidos para este primeiro modelo serão:

• random_state: para repetibilidade do treinamento do modelo. Será utilizado na busca pelos hiperparâmetros também, sempre como mesmo valor definido na variável seed.

• max_depth: será definido como 4, pois o padrão do modelo é 3 de profundidade máxima para as árvores, deixando o modelo muito simples.

• scale_pos_weight: usado para definir o peso da classe 1 no treinamento do modelo, pois temos classes desbalanceadas.

# Criação e treinamento do modelo XGBoost
xgb = XGBClassifier(max_depth=4,
                    scale_pos_weight=1.58,
                    random_state=seed)

xgb.fit(X_train, y_train)

XGBClassifier(base_score=None, booster=None, callbacks=None,
              colsample_bylevel=None, colsample_bynode=None,
              colsample_bytree=None, early_stopping_rounds=None,
              enable_categorical=False, eval_metric=None, feature_types=None,
              gamma=None, gpu_id=None, grow_policy=None, importance_type=None,
              interaction_constraints=None, learning_rate=None, max_bin=None,
              max_cat_threshold=None, max_cat_to_onehot=None,
              max_delta_step=None, max_depth=4, max_leaves=None,
              min_child_weight=None, missing=nan, monotone_constraints=None,
              n_estimators=100, n_jobs=None, num_parallel_tree=None,
              predictor=None, random_state=10, ...)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# Matriz de confusão
plot_confusion_matrix(xgb, X_test, y_test)

              precision    recall  f1-score   support

           0      0.823     0.765     0.793      4695
           1      0.695     0.764     0.728      3284

    accuracy                          0.765      7979
   macro avg      0.759     0.765     0.760      7979
weighted avg      0.770     0.765     0.766      7979

A matriz obtida para o modelo XGBoost apresentou diagonal equilibrada em ambas as classes, com 77% de acurácia.

# Curva ROC
plot_roc_curve(xgb, X_train, X_test, y_train, y_test)

Como a métrica AUC possui valores próximos para o conjunto de treino e de teste, 0,887 e 0,842 respectivamente, pode-se dizer que há apenas um pouco de overfitting, não sendo algo de grande preocupação.

# Importância das features
plot_feat_importances(xgb, feat_cols)

As features mais importantes nesta visualização são RECNENHUM, EC, CATEATEND e CIRURGIA.

# Importância das features pelos valores SHAP
plot_shap_values(xgb, X_train, feat_cols)

A coluna EC foi a mais importante, com isso, os valores mais altos desta variável, mostrados em rosa, influenciaram mais o modelo na previsão da classe 1 (óbito por câncer). Já os valores mais baixos desta coluna, em azul, tem mais peso para previsão ser da classe 0. Este comportamento faz sentido, pois quanto mais alto o estágio, maior é a extensão do câncer, assim menor a chance de sobrevivência.

O raciocínio para analisar as outras colunas é o mesmo utilizado para o estadiamento clínico.

Optuna

Para fazer a busca pelos melhores hiperparâmetros, será utilizado a biblioteca Optuna, definindo o intervalo para os parâmetros do modelo a serem buscados.

# Folds com a mesma proporção das classes
skf = StratifiedKFold(10, shuffle=True, random_state=seed)

# Função com o modelos e seus parâmetros, que terá sua métrica maximizada
def objective(trial):

    n_estimators = trial.suggest_int('n_estimators', 50, 250)
    max_depth = trial.suggest_int('max_depth', 3, 18)
    learning_rate = trial.suggest_float('learning_rate', 0.05, 0.2, step=0.05)
    gamma = trial.suggest_float('gamma', 0.0, 0.3, step=0.1)
    min_child_weight = trial.suggest_int('min_child_weight', 1, 7)
    colsample_bytree = trial.suggest_float('colsample_bytree', 0.3, 0.7, step=0.1)

    cls = XGBClassifier(n_estimators=n_estimators,
                        max_depth=max_depth,
                        learning_rate=learning_rate,
                        gamma=gamma,
                        min_child_weight=min_child_weight,
                        colsample_bytree=colsample_bytree,
                        random_state=seed)

    return cross_val_score(cls, X_train, y_train,
                           cv=skf, scoring='balanced_accuracy').mean()

# Criação do estudo e procura pelos hiperparâmetros
studyXGB = optuna.create_study(direction='maximize', sampler=RandomSampler(seed))
studyXGB.optimize(objective, n_trials=100)

# Melhor tentativa
studyXGB.best_trial

FrozenTrial(number=61, state=TrialState.COMPLETE, values=[0.7643089701876458], datetime_start=datetime.datetime(2023, 4, 12, 2, 1, 32, 43835), datetime_complete=datetime.datetime(2023, 4, 12, 2, 2, 5, 449827), params={'n_estimators': 205, 'max_depth': 5, 'learning_rate': 0.15000000000000002, 'gamma': 0.3, 'min_child_weight': 2, 'colsample_bytree': 0.5}, user_attrs={}, system_attrs={}, intermediate_values={}, distributions={'n_estimators': IntDistribution(high=250, log=False, low=50, step=1), 'max_depth': IntDistribution(high=18, log=False, low=3, step=1), 'learning_rate': FloatDistribution(high=0.2, log=False, low=0.05, step=0.05), 'gamma': FloatDistribution(high=0.3, log=False, low=0.0, step=0.1), 'min_child_weight': IntDistribution(high=7, log=False, low=1, step=1), 'colsample_bytree': FloatDistribution(high=0.7, log=False, low=0.3, step=0.1)}, trial_id=61, value=None)

# Melhores parâmetros
studyXGB.best_params

{'n_estimators': 205,
 'max_depth': 5,
 'learning_rate': 0.15000000000000002,
 'gamma': 0.3,
 'min_child_weight': 2,
 'colsample_bytree': 0.5}

plot_optimization_history(studyXGB).show()

# Modelo com os melhores parâmetros
params = studyXGB.best_params
params['random_state'] = seed
params['scale_pos_weight'] = 1.575

xgb_optuna = XGBClassifier()
xgb_optuna.set_params(**params)

xgb_optuna.fit(X_train, y_train)

XGBClassifier(base_score=None, booster=None, callbacks=None,
              colsample_bylevel=None, colsample_bynode=None,
              colsample_bytree=0.5, early_stopping_rounds=None,
              enable_categorical=False, eval_metric=None, feature_types=None,
              gamma=0.3, gpu_id=None, grow_policy=None, importance_type=None,
              interaction_constraints=None, learning_rate=0.15000000000000002,
              max_bin=None, max_cat_threshold=None, max_cat_to_onehot=None,
              max_delta_step=None, max_depth=5, max_leaves=None,
              min_child_weight=2, missing=nan, monotone_constraints=None,
              n_estimators=205, n_jobs=None, num_parallel_tree=None,
              predictor=None, random_state=10, ...)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# Matriz de confusão do modelo XGBoost otimizado
plot_confusion_matrix(xgb_optuna, X_test, y_test)

              precision    recall  f1-score   support

           0      0.828     0.770     0.798      4695
           1      0.701     0.771     0.735      3284

    accuracy                          0.771      7979
   macro avg      0.765     0.771     0.766      7979
weighted avg      0.776     0.771     0.772      7979

Após a escolha dos hiperparâmetros, a acurácia de ambos os modelos ficou em torno de 77%.

# Curva ROC do modelo otimizado
plot_roc_curve(xgb_optuna, X_train, X_test, y_train, y_test)

A curva ROC mostra que o modelo possui um pouco de overfitting, pois para o conjunto de treino temos AUC = 0,908 e para o teste AUC = 0,845, essa diferença caracteriza o problema.

# Importância das features pelos valores SHAP, para o modelo com os melhores hiperparâmetros
plot_shap_values(xgb_optuna, X_train, feat_cols)

ntree_limit is deprecated, use `iteration_range` or model slicing instead.
No data for colormapping provided via 'c'. Parameters 'vmin', 'vmax' will be ignored

ROCs

# Treino
roc_together(X_train, y_train, nb, rf_optuna, xgb_optuna)

# Teste
roc_together(X_test, y_test, nb, rf_optuna, xgb_optuna)

RN

Importação das bibliotecas e funções

import tensorflow as tf
from tensorflow import keras

import matplotlib.pyplot as plt

from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Dense, Dropout, Add, Input, Activation
from tensorflow.keras.optimizers import Adam

from tensorflow.keras.regularizers import l2
from tensorflow.keras.callbacks import EarlyStopping

# Definição de cores para gráficos
colors = plt.rcParams['axes.prop_cycle'].by_key()['color']

def plot_metrics(history):
    """Plot metrics after training the RNA.

    :param history: RNA training history.

    :return: no value
    :rtype: none
    """
    metrics = ['loss', 'accuracy', 'precision', 'recall']
    plt.figure(figsize=(12,8))
    for n, metric in enumerate(metrics):
        name = metric.replace("_"," ").capitalize()
        plt.subplot(2, 2, n + 1)
        plt.plot(history.epoch, history.history[metric], color=colors[0], label='Train')
        plt.plot(history.epoch, history.history['val_'+ metric],
                 color=colors[0], linestyle="--", label='Val')
        plt.xlabel('Epoch')
        plt.ylabel(name)
        if metric == 'loss':
            plt.ylim([0, plt.ylim()[1]])
        if metric == 'accuracy':
            plt.ylim([0.7, 1])
        else:
            plt.ylim([0, 1])
        plt.legend()

Criação e treinamento da RNA Complexa

neg, pos = np.bincount(y_train)
total = neg + pos
print(f'Exemplos:\n Total: {total}\n Positivos: {pos} ({100*pos/total:.2f}% do total)')

# Cálculo dos pesos das duas classe
weight_for_0 = (1 / neg)*(total)/2.0
weight_for_1 = (1 / pos)*(total)/2.0

# Dicionário de pesos das classes para treinamento
class_weight = {0: weight_for_0, 1: weight_for_1}
print('Peso da classe 0: {:.2f}'.format(weight_for_0))
print('Peso da classe 1: {:.2f}'.format(weight_for_1))

Exemplos:
 Total: 23937
 Positivos: 9852 (41.16% do total)
Peso da classe 0: 0.85
Peso da classe 1: 1.21

input_shape = X_train.shape[1:]
input_features = Input(shape=input_shape, name='input_features')

x1 = Dense(128, activation='tanh', kernel_regularizer=l2())(input_features)
x2 = Dense(128, activation='selu', kernel_regularizer=l2())(input_features)
x3 = Dense(128, activation='sigmoid', kernel_regularizer=l2())(input_features)

from tensorflow.keras.layers import Concatenate
x_concat = Concatenate()([x1, x2, x3, input_features])

x4 = Dense(32, activation='relu', kernel_regularizer=l2())(x_concat)
out = Dense(1, activation='sigmoid', name='out_dense')(x4)

model = keras.Model(inputs=[input_features],
                    outputs=[out])

model.summary()

Model: "model"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to
==================================================================================================
 input_features (InputLayer)    [(None, 25)]         0           []

 dense (Dense)                  (None, 128)          3328        ['input_features[0][0]']

 dense_1 (Dense)                (None, 128)          3328        ['input_features[0][0]']

 dense_2 (Dense)                (None, 128)          3328        ['input_features[0][0]']

 concatenate (Concatenate)      (None, 409)          0           ['dense[0][0]',
                                                                  'dense_1[0][0]',
                                                                  'dense_2[0][0]',
                                                                  'input_features[0][0]']

 dense_3 (Dense)                (None, 32)           13120       ['concatenate[0][0]']

 out_dense (Dense)              (None, 1)            33          ['dense_3[0][0]']

==================================================================================================
Total params: 23,137
Trainable params: 23,137
Non-trainable params: 0
__________________________________________________________________________________________________

keras.utils.plot_model(model, show_shapes=True)

from tensorflow.keras.callbacks import EarlyStopping

# Define métricas
METRICS = [keras.metrics.BinaryAccuracy(name='accuracy'),
           keras.metrics.Precision(name='precision'),
           keras.metrics.Recall(name='recall'),
           keras.metrics.AUC(name='auc')]

call_es = EarlyStopping(monitor='val_loss', patience=20, restore_best_weights=True)

adam = Adam(learning_rate=0.001)
model.compile(optimizer=adam, loss='binary_crossentropy',
              metrics=METRICS)

history = model.fit(X_train, y_train, epochs=50,
                    class_weight=class_weight,
                    verbose=2, batch_size=32,
                    validation_data=(X_test, y_test),
                    callbacks=[call_es])

Epoch 1/50
749/749 - 4s - loss: 0.8618 - accuracy: 0.7352 - precision: 0.6591 - recall: 0.7387 - auc: 0.8048 - val_loss: 0.5659 - val_accuracy: 0.7424 - val_precision: 0.6691 - val_recall: 0.7403 - val_auc: 0.8121 - 4s/epoch - 5ms/step
Epoch 2/50
749/749 - 2s - loss: 0.5535 - accuracy: 0.7409 - precision: 0.6676 - recall: 0.7379 - auc: 0.8120 - val_loss: 0.5461 - val_accuracy: 0.7422 - val_precision: 0.6704 - val_recall: 0.7351 - val_auc: 0.8110 - 2s/epoch - 3ms/step
Epoch 3/50
749/749 - 2s - loss: 0.5464 - accuracy: 0.7420 - precision: 0.6677 - recall: 0.7427 - auc: 0.8128 - val_loss: 0.5326 - val_accuracy: 0.7501 - val_precision: 0.7014 - val_recall: 0.6839 - val_auc: 0.8146 - 2s/epoch - 3ms/step
Epoch 4/50
749/749 - 2s - loss: 0.5435 - accuracy: 0.7435 - precision: 0.6704 - recall: 0.7413 - auc: 0.8137 - val_loss: 0.5366 - val_accuracy: 0.7431 - val_precision: 0.7247 - val_recall: 0.6060 - val_auc: 0.8113 - 2s/epoch - 3ms/step
Epoch 5/50
749/749 - 3s - loss: 0.5426 - accuracy: 0.7461 - precision: 0.6747 - recall: 0.7397 - auc: 0.8151 - val_loss: 0.5340 - val_accuracy: 0.7475 - val_precision: 0.6820 - val_recall: 0.7241 - val_auc: 0.8165 - 3s/epoch - 5ms/step
Epoch 6/50
749/749 - 3s - loss: 0.5405 - accuracy: 0.7482 - precision: 0.6754 - recall: 0.7473 - auc: 0.8165 - val_loss: 0.5477 - val_accuracy: 0.7413 - val_precision: 0.6560 - val_recall: 0.7811 - val_auc: 0.8201 - 3s/epoch - 4ms/step
Epoch 7/50
749/749 - 1s - loss: 0.5399 - accuracy: 0.7484 - precision: 0.6765 - recall: 0.7449 - auc: 0.8171 - val_loss: 0.5476 - val_accuracy: 0.7374 - val_precision: 0.6492 - val_recall: 0.7878 - val_auc: 0.8202 - 1s/epoch - 2ms/step
Epoch 8/50
749/749 - 1s - loss: 0.5384 - accuracy: 0.7475 - precision: 0.6766 - recall: 0.7408 - auc: 0.8176 - val_loss: 0.5310 - val_accuracy: 0.7512 - val_precision: 0.6816 - val_recall: 0.7424 - val_auc: 0.8212 - 1s/epoch - 2ms/step
Epoch 9/50
749/749 - 1s - loss: 0.5370 - accuracy: 0.7471 - precision: 0.6754 - recall: 0.7424 - auc: 0.8193 - val_loss: 0.5320 - val_accuracy: 0.7466 - val_precision: 0.6829 - val_recall: 0.7174 - val_auc: 0.8180 - 1s/epoch - 2ms/step
Epoch 10/50
749/749 - 1s - loss: 0.5357 - accuracy: 0.7500 - precision: 0.6790 - recall: 0.7443 - auc: 0.8204 - val_loss: 0.5289 - val_accuracy: 0.7522 - val_precision: 0.7172 - val_recall: 0.6571 - val_auc: 0.8191 - 1s/epoch - 2ms/step
Epoch 11/50
749/749 - 1s - loss: 0.5343 - accuracy: 0.7483 - precision: 0.6765 - recall: 0.7443 - auc: 0.8210 - val_loss: 0.5287 - val_accuracy: 0.7531 - val_precision: 0.7035 - val_recall: 0.6915 - val_auc: 0.8194 - 1s/epoch - 2ms/step
Epoch 12/50
749/749 - 2s - loss: 0.5349 - accuracy: 0.7493 - precision: 0.6791 - recall: 0.7409 - auc: 0.8205 - val_loss: 0.5336 - val_accuracy: 0.7491 - val_precision: 0.6717 - val_recall: 0.7637 - val_auc: 0.8221 - 2s/epoch - 2ms/step
Epoch 13/50
749/749 - 2s - loss: 0.5312 - accuracy: 0.7515 - precision: 0.6803 - recall: 0.7476 - auc: 0.8229 - val_loss: 0.5411 - val_accuracy: 0.7433 - val_precision: 0.6580 - val_recall: 0.7838 - val_auc: 0.8232 - 2s/epoch - 2ms/step
Epoch 14/50
749/749 - 1s - loss: 0.5309 - accuracy: 0.7513 - precision: 0.6822 - recall: 0.7407 - auc: 0.8234 - val_loss: 0.5455 - val_accuracy: 0.7432 - val_precision: 0.6641 - val_recall: 0.7610 - val_auc: 0.8187 - 1s/epoch - 2ms/step
Epoch 15/50
749/749 - 1s - loss: 0.5298 - accuracy: 0.7515 - precision: 0.6807 - recall: 0.7460 - auc: 0.8234 - val_loss: 0.5600 - val_accuracy: 0.7305 - val_precision: 0.6346 - val_recall: 0.8139 - val_auc: 0.8231 - 1s/epoch - 2ms/step
Epoch 16/50
749/749 - 1s - loss: 0.5294 - accuracy: 0.7531 - precision: 0.6828 - recall: 0.7472 - auc: 0.8245 - val_loss: 0.5378 - val_accuracy: 0.7427 - val_precision: 0.6606 - val_recall: 0.7710 - val_auc: 0.8224 - 1s/epoch - 2ms/step
Epoch 17/50
749/749 - 1s - loss: 0.5314 - accuracy: 0.7519 - precision: 0.6803 - recall: 0.7493 - auc: 0.8233 - val_loss: 0.5197 - val_accuracy: 0.7579 - val_precision: 0.7060 - val_recall: 0.7055 - val_auc: 0.8254 - 1s/epoch - 2ms/step
Epoch 18/50
749/749 - 1s - loss: 0.5280 - accuracy: 0.7532 - precision: 0.6828 - recall: 0.7477 - auc: 0.8253 - val_loss: 0.5242 - val_accuracy: 0.7496 - val_precision: 0.6895 - val_recall: 0.7125 - val_auc: 0.8225 - 1s/epoch - 2ms/step
Epoch 19/50
749/749 - 1s - loss: 0.5276 - accuracy: 0.7539 - precision: 0.6844 - recall: 0.7463 - auc: 0.8251 - val_loss: 0.5317 - val_accuracy: 0.7492 - val_precision: 0.6703 - val_recall: 0.7689 - val_auc: 0.8250 - 1s/epoch - 2ms/step
Epoch 20/50
749/749 - 1s - loss: 0.5265 - accuracy: 0.7538 - precision: 0.6837 - recall: 0.7477 - auc: 0.8262 - val_loss: 0.5241 - val_accuracy: 0.7527 - val_precision: 0.6821 - val_recall: 0.7476 - val_auc: 0.8251 - 1s/epoch - 2ms/step
Epoch 21/50
749/749 - 2s - loss: 0.5270 - accuracy: 0.7544 - precision: 0.6855 - recall: 0.7452 - auc: 0.8255 - val_loss: 0.5359 - val_accuracy: 0.7452 - val_precision: 0.6626 - val_recall: 0.7762 - val_auc: 0.8254 - 2s/epoch - 3ms/step
Epoch 22/50
749/749 - 2s - loss: 0.5249 - accuracy: 0.7550 - precision: 0.6857 - recall: 0.7475 - auc: 0.8271 - val_loss: 0.5413 - val_accuracy: 0.7421 - val_precision: 0.6546 - val_recall: 0.7905 - val_auc: 0.8270 - 2s/epoch - 2ms/step
Epoch 23/50
749/749 - 1s - loss: 0.5265 - accuracy: 0.7554 - precision: 0.6863 - recall: 0.7472 - auc: 0.8268 - val_loss: 0.5234 - val_accuracy: 0.7537 - val_precision: 0.6892 - val_recall: 0.7314 - val_auc: 0.8248 - 1s/epoch - 2ms/step
Epoch 24/50
749/749 - 1s - loss: 0.5246 - accuracy: 0.7547 - precision: 0.6862 - recall: 0.7446 - auc: 0.8272 - val_loss: 0.5241 - val_accuracy: 0.7536 - val_precision: 0.6852 - val_recall: 0.7424 - val_auc: 0.8262 - 1s/epoch - 2ms/step
Epoch 25/50
749/749 - 1s - loss: 0.5238 - accuracy: 0.7554 - precision: 0.6867 - recall: 0.7462 - auc: 0.8282 - val_loss: 0.5288 - val_accuracy: 0.7490 - val_precision: 0.6758 - val_recall: 0.7497 - val_auc: 0.8261 - 1s/epoch - 2ms/step
Epoch 26/50
749/749 - 1s - loss: 0.5239 - accuracy: 0.7563 - precision: 0.6873 - recall: 0.7482 - auc: 0.8278 - val_loss: 0.5309 - val_accuracy: 0.7477 - val_precision: 0.6688 - val_recall: 0.7667 - val_auc: 0.8264 - 1s/epoch - 2ms/step
Epoch 27/50
749/749 - 1s - loss: 0.5225 - accuracy: 0.7572 - precision: 0.6889 - recall: 0.7478 - auc: 0.8284 - val_loss: 0.5321 - val_accuracy: 0.7448 - val_precision: 0.6684 - val_recall: 0.7543 - val_auc: 0.8240 - 1s/epoch - 2ms/step
Epoch 28/50
749/749 - 2s - loss: 0.5220 - accuracy: 0.7558 - precision: 0.6876 - recall: 0.7452 - auc: 0.8289 - val_loss: 0.5230 - val_accuracy: 0.7570 - val_precision: 0.7006 - val_recall: 0.7153 - val_auc: 0.8231 - 2s/epoch - 2ms/step
Epoch 29/50
749/749 - 3s - loss: 0.5234 - accuracy: 0.7554 - precision: 0.6879 - recall: 0.7427 - auc: 0.8284 - val_loss: 0.5280 - val_accuracy: 0.7477 - val_precision: 0.6722 - val_recall: 0.7555 - val_auc: 0.8265 - 3s/epoch - 4ms/step
Epoch 30/50
749/749 - 2s - loss: 0.5218 - accuracy: 0.7574 - precision: 0.6897 - recall: 0.7464 - auc: 0.8292 - val_loss: 0.5284 - val_accuracy: 0.7475 - val_precision: 0.6726 - val_recall: 0.7530 - val_auc: 0.8258 - 2s/epoch - 2ms/step
Epoch 31/50
749/749 - 1s - loss: 0.5221 - accuracy: 0.7562 - precision: 0.6886 - recall: 0.7441 - auc: 0.8291 - val_loss: 0.5325 - val_accuracy: 0.7467 - val_precision: 0.6714 - val_recall: 0.7533 - val_auc: 0.8230 - 1s/epoch - 2ms/step
Epoch 32/50
749/749 - 1s - loss: 0.5217 - accuracy: 0.7573 - precision: 0.6902 - recall: 0.7444 - auc: 0.8293 - val_loss: 0.5245 - val_accuracy: 0.7537 - val_precision: 0.6832 - val_recall: 0.7488 - val_auc: 0.8266 - 1s/epoch - 2ms/step
Epoch 33/50
749/749 - 1s - loss: 0.5197 - accuracy: 0.7583 - precision: 0.6912 - recall: 0.7462 - auc: 0.8305 - val_loss: 0.5181 - val_accuracy: 0.7572 - val_precision: 0.6993 - val_recall: 0.7195 - val_auc: 0.8274 - 1s/epoch - 2ms/step
Epoch 34/50
749/749 - 1s - loss: 0.5205 - accuracy: 0.7580 - precision: 0.6918 - recall: 0.7429 - auc: 0.8303 - val_loss: 0.5356 - val_accuracy: 0.7433 - val_precision: 0.6606 - val_recall: 0.7741 - val_auc: 0.8260 - 1s/epoch - 2ms/step
Epoch 35/50
749/749 - 1s - loss: 0.5210 - accuracy: 0.7576 - precision: 0.6903 - recall: 0.7454 - auc: 0.8300 - val_loss: 0.5211 - val_accuracy: 0.7549 - val_precision: 0.6928 - val_recall: 0.7266 - val_auc: 0.8273 - 1s/epoch - 2ms/step
Epoch 36/50
749/749 - 1s - loss: 0.5188 - accuracy: 0.7581 - precision: 0.6919 - recall: 0.7430 - auc: 0.8310 - val_loss: 0.5298 - val_accuracy: 0.7491 - val_precision: 0.6721 - val_recall: 0.7622 - val_auc: 0.8246 - 1s/epoch - 2ms/step
Epoch 37/50
749/749 - 2s - loss: 0.5193 - accuracy: 0.7594 - precision: 0.6934 - recall: 0.7446 - auc: 0.8309 - val_loss: 0.5266 - val_accuracy: 0.7520 - val_precision: 0.6825 - val_recall: 0.7430 - val_auc: 0.8255 - 2s/epoch - 2ms/step
Epoch 38/50
749/749 - 2s - loss: 0.5208 - accuracy: 0.7599 - precision: 0.6935 - recall: 0.7463 - auc: 0.8303 - val_loss: 0.5220 - val_accuracy: 0.7570 - val_precision: 0.6886 - val_recall: 0.7476 - val_auc: 0.8284 - 2s/epoch - 3ms/step
Epoch 39/50
749/749 - 1s - loss: 0.5199 - accuracy: 0.7581 - precision: 0.6907 - recall: 0.7468 - auc: 0.8307 - val_loss: 0.5250 - val_accuracy: 0.7552 - val_precision: 0.6874 - val_recall: 0.7433 - val_auc: 0.8266 - 1s/epoch - 2ms/step
Epoch 40/50
749/749 - 1s - loss: 0.5194 - accuracy: 0.7589 - precision: 0.6920 - recall: 0.7463 - auc: 0.8309 - val_loss: 0.5193 - val_accuracy: 0.7586 - val_precision: 0.6949 - val_recall: 0.7372 - val_auc: 0.8294 - 1s/epoch - 2ms/step
Epoch 41/50
749/749 - 1s - loss: 0.5215 - accuracy: 0.7588 - precision: 0.6947 - recall: 0.7387 - auc: 0.8297 - val_loss: 0.5215 - val_accuracy: 0.7512 - val_precision: 0.6790 - val_recall: 0.7503 - val_auc: 0.8288 - 1s/epoch - 2ms/step
Epoch 42/50
749/749 - 1s - loss: 0.5185 - accuracy: 0.7586 - precision: 0.6914 - recall: 0.7469 - auc: 0.8312 - val_loss: 0.5243 - val_accuracy: 0.7542 - val_precision: 0.6825 - val_recall: 0.7533 - val_auc: 0.8286 - 1s/epoch - 2ms/step
Epoch 43/50
749/749 - 2s - loss: 0.5191 - accuracy: 0.7580 - precision: 0.6908 - recall: 0.7457 - auc: 0.8309 - val_loss: 0.5191 - val_accuracy: 0.7571 - val_precision: 0.6951 - val_recall: 0.7302 - val_auc: 0.8279 - 2s/epoch - 2ms/step
Epoch 44/50
749/749 - 2s - loss: 0.5181 - accuracy: 0.7595 - precision: 0.6932 - recall: 0.7454 - auc: 0.8317 - val_loss: 0.5347 - val_accuracy: 0.7512 - val_precision: 0.6755 - val_recall: 0.7613 - val_auc: 0.8242 - 2s/epoch - 2ms/step
Epoch 45/50
749/749 - 2s - loss: 0.5187 - accuracy: 0.7585 - precision: 0.6921 - recall: 0.7446 - auc: 0.8312 - val_loss: 0.5187 - val_accuracy: 0.7546 - val_precision: 0.6879 - val_recall: 0.7390 - val_auc: 0.8287 - 2s/epoch - 2ms/step
Epoch 46/50
749/749 - 2s - loss: 0.5175 - accuracy: 0.7595 - precision: 0.6928 - recall: 0.7469 - auc: 0.8319 - val_loss: 0.5221 - val_accuracy: 0.7549 - val_precision: 0.6904 - val_recall: 0.7333 - val_auc: 0.8277 - 2s/epoch - 3ms/step
Epoch 47/50
749/749 - 1s - loss: 0.5175 - accuracy: 0.7607 - precision: 0.6949 - recall: 0.7461 - auc: 0.8320 - val_loss: 0.5236 - val_accuracy: 0.7531 - val_precision: 0.6819 - val_recall: 0.7500 - val_auc: 0.8289 - 1s/epoch - 2ms/step
Epoch 48/50
749/749 - 1s - loss: 0.5188 - accuracy: 0.7597 - precision: 0.6949 - recall: 0.7419 - auc: 0.8313 - val_loss: 0.5255 - val_accuracy: 0.7502 - val_precision: 0.6780 - val_recall: 0.7488 - val_auc: 0.8255 - 1s/epoch - 2ms/step
Epoch 49/50
749/749 - 1s - loss: 0.5178 - accuracy: 0.7593 - precision: 0.6936 - recall: 0.7438 - auc: 0.8318 - val_loss: 0.5304 - val_accuracy: 0.7515 - val_precision: 0.6786 - val_recall: 0.7527 - val_auc: 0.8263 - 1s/epoch - 2ms/step
Epoch 50/50
749/749 - 1s - loss: 0.5169 - accuracy: 0.7602 - precision: 0.6944 - recall: 0.7455 - auc: 0.8322 - val_loss: 0.5167 - val_accuracy: 0.7591 - val_precision: 0.6951 - val_recall: 0.7387 - val_auc: 0.8305 - 1s/epoch - 2ms/step

Resultados

plot_metrics(history)

train_pred = model.predict(X_train)
test_pred = model.predict(X_test)

print('Número de exemplos positivos do conjunto de teste =', len(y_test[y_test > 0.9]))
results = model.evaluate(X_test, y_test, verbose=0)

for name, value in zip(model.metrics_names, results):
    print(f'{name}: {value:.4f}')

749/749 [==============================] - 1s 958us/step
250/250 [==============================] - 0s 1ms/step
Número de exemplos positivos do conjunto de teste = 3284
loss: 0.5167
accuracy: 0.7591
precision: 0.6951
recall: 0.7387
auc: 0.8305

precision = results[2]
recall = results[3]
F1 = 2*precision*recall/(precision + recall)
print(f'Pontuação F1 = {F1:.4f}')

Pontuação F1 = 0.7163

ConfusionMatrixDisplay.from_predictions(y_test, np.round(test_pred),
                                        normalize='true', cmap='Blues',
                                        values_format='.3f')
plt.show()

from sklearn.metrics import balanced_accuracy_score

balanced_accuracy_score(y_test, np.round(test_pred))

0.7560545919869662

fp_train, tp_train, _ = roc_curve(y_train, train_pred)
fp_test, tp_test, _ = roc_curve(y_test, test_pred)
plt.figure(figsize=(8, 6))
plt.plot(100*fp_train, 100*tp_train, 'b', label='Dados treinamento')
plt.plot(100*fp_test, 100*tp_test, 'r', label='Dados teste')
plt.xlabel('Positivos falsos [%]')
plt.ylabel('Positivos verdadeiros [%]')
plt.xlim([0,100])
plt.ylim([0,100])
plt.grid(True)
plt.legend()
plt.show()

custo_e_metricas_train = model.evaluate(X_train, y_train)
# custo_e_metricas_val = rna_reg.evaluate(X_val_norm, y_val)
custo_e_metricas_test = model.evaluate(X_test, y_test)

749/749 [==============================] - 1s 1ms/step - loss: 0.5104 - accuracy: 0.7650 - precision: 0.7074 - recall: 0.7314 - auc: 0.8347
250/250 [==============================] - 0s 2ms/step - loss: 0.5167 - accuracy: 0.7591 - precision: 0.6951 - recall: 0.7387 - auc: 0.8305

Criação e treinamento da RNA Residual

# Função que cria o bloco para a RN residual
def bloco_residual(x, n):

    z1 = Dense(n, activation='relu')(x)
    z2 = Dense(n)(z1)
    sum = Add()([x, z2])
    a2 = Activation('relu')(sum)

    return a2

# Definição da entrada
input_shape = X_train.shape[1:]
input_features = Input(shape=input_shape)

X1 = Dense(64, activation='relu')(input_features)

X2 = bloco_residual(X1, 64)

X3 = Dense(64, activation='relu')(X2)

X4 = bloco_residual(X3, 64)

X5 = Dense(16, activation='relu')(X4)

Y = Dense(units=1, activation='sigmoid')(X5)

# Criação da RNA
rna = Model(input_features, Y)

# Mostra resumo da RNA
rna.summary()

Model: "model_1"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to
==================================================================================================
 input_1 (InputLayer)           [(None, 25)]         0           []

 dense_4 (Dense)                (None, 64)           1664        ['input_1[0][0]']

 dense_5 (Dense)                (None, 64)           4160        ['dense_4[0][0]']

 dense_6 (Dense)                (None, 64)           4160        ['dense_5[0][0]']

 add (Add)                      (None, 64)           0           ['dense_4[0][0]',
                                                                  'dense_6[0][0]']

 activation (Activation)        (None, 64)           0           ['add[0][0]']

 dense_7 (Dense)                (None, 64)           4160        ['activation[0][0]']

 dense_8 (Dense)                (None, 64)           4160        ['dense_7[0][0]']

 dense_9 (Dense)                (None, 64)           4160        ['dense_8[0][0]']

 add_1 (Add)                    (None, 64)           0           ['dense_7[0][0]',
                                                                  'dense_9[0][0]']

 activation_1 (Activation)      (None, 64)           0           ['add_1[0][0]']

 dense_10 (Dense)               (None, 16)           1040        ['activation_1[0][0]']

 dense_11 (Dense)               (None, 1)            17          ['dense_10[0][0]']

==================================================================================================
Total params: 23,521
Trainable params: 23,521
Non-trainable params: 0
__________________________________________________________________________________________________

keras.utils.plot_model(rna, show_shapes=True)

from tensorflow.keras.callbacks import EarlyStopping

# Define métricas
METRICS = [keras.metrics.BinaryAccuracy(name='accuracy'),
           keras.metrics.Precision(name='precision'),
           keras.metrics.Recall(name='recall'),
           keras.metrics.AUC(name='auc')]

call_es = EarlyStopping(monitor='val_loss', patience=20,
                        restore_best_weights=True)

adam = Adam(learning_rate=0.001)
rna.compile(optimizer=adam, loss='binary_crossentropy',
              metrics=METRICS)

history = rna.fit(X_train, y_train, epochs=50,
                    class_weight=class_weight,
                    verbose=2, batch_size=32,
                    validation_data=(X_test, y_test),
                    callbacks=[call_es])

Epoch 1/50
749/749 - 3s - loss: 0.5443 - accuracy: 0.7289 - precision: 0.6524 - recall: 0.7305 - auc: 0.8009 - val_loss: 0.5188 - val_accuracy: 0.7440 - val_precision: 0.6819 - val_recall: 0.7083 - val_auc: 0.8179 - 3s/epoch - 4ms/step
Epoch 2/50
749/749 - 2s - loss: 0.5139 - accuracy: 0.7531 - precision: 0.6840 - recall: 0.7436 - auc: 0.8261 - val_loss: 0.5108 - val_accuracy: 0.7559 - val_precision: 0.7080 - val_recall: 0.6924 - val_auc: 0.8225 - 2s/epoch - 3ms/step
Epoch 3/50
749/749 - 2s - loss: 0.5093 - accuracy: 0.7594 - precision: 0.6935 - recall: 0.7444 - auc: 0.8294 - val_loss: 0.5188 - val_accuracy: 0.7511 - val_precision: 0.6937 - val_recall: 0.7077 - val_auc: 0.8227 - 2s/epoch - 3ms/step
Epoch 4/50
749/749 - 1s - loss: 0.5017 - accuracy: 0.7619 - precision: 0.6982 - recall: 0.7426 - auc: 0.8351 - val_loss: 0.5082 - val_accuracy: 0.7515 - val_precision: 0.6995 - val_recall: 0.6946 - val_auc: 0.8242 - 1s/epoch - 2ms/step
Epoch 5/50
749/749 - 1s - loss: 0.4954 - accuracy: 0.7666 - precision: 0.7035 - recall: 0.7481 - auc: 0.8397 - val_loss: 0.5096 - val_accuracy: 0.7554 - val_precision: 0.6928 - val_recall: 0.7287 - val_auc: 0.8285 - 1s/epoch - 2ms/step
Epoch 6/50
749/749 - 1s - loss: 0.4902 - accuracy: 0.7694 - precision: 0.7067 - recall: 0.7517 - auc: 0.8435 - val_loss: 0.5427 - val_accuracy: 0.7418 - val_precision: 0.6602 - val_recall: 0.7680 - val_auc: 0.8195 - 1s/epoch - 2ms/step
Epoch 7/50
749/749 - 2s - loss: 0.4839 - accuracy: 0.7722 - precision: 0.7102 - recall: 0.7544 - auc: 0.8480 - val_loss: 0.5038 - val_accuracy: 0.7597 - val_precision: 0.7151 - val_recall: 0.6918 - val_auc: 0.8282 - 2s/epoch - 2ms/step
Epoch 8/50
749/749 - 1s - loss: 0.4799 - accuracy: 0.7742 - precision: 0.7119 - recall: 0.7580 - auc: 0.8509 - val_loss: 0.5149 - val_accuracy: 0.7497 - val_precision: 0.6760 - val_recall: 0.7527 - val_auc: 0.8274 - 1s/epoch - 2ms/step
Epoch 9/50
749/749 - 1s - loss: 0.4727 - accuracy: 0.7774 - precision: 0.7134 - recall: 0.7675 - auc: 0.8555 - val_loss: 0.5340 - val_accuracy: 0.7421 - val_precision: 0.6710 - val_recall: 0.7323 - val_auc: 0.8185 - 1s/epoch - 2ms/step
Epoch 10/50
749/749 - 2s - loss: 0.4698 - accuracy: 0.7764 - precision: 0.7115 - recall: 0.7684 - auc: 0.8578 - val_loss: 0.5105 - val_accuracy: 0.7531 - val_precision: 0.6959 - val_recall: 0.7107 - val_auc: 0.8226 - 2s/epoch - 2ms/step
Epoch 11/50
749/749 - 2s - loss: 0.4644 - accuracy: 0.7822 - precision: 0.7200 - recall: 0.7706 - auc: 0.8615 - val_loss: 0.5178 - val_accuracy: 0.7540 - val_precision: 0.6920 - val_recall: 0.7250 - val_auc: 0.8236 - 2s/epoch - 3ms/step
Epoch 12/50
749/749 - 2s - loss: 0.4561 - accuracy: 0.7849 - precision: 0.7226 - recall: 0.7746 - auc: 0.8669 - val_loss: 0.5277 - val_accuracy: 0.7451 - val_precision: 0.6688 - val_recall: 0.7540 - val_auc: 0.8195 - 2s/epoch - 2ms/step
Epoch 13/50
749/749 - 1s - loss: 0.4499 - accuracy: 0.7889 - precision: 0.7273 - recall: 0.7792 - auc: 0.8704 - val_loss: 0.5375 - val_accuracy: 0.7461 - val_precision: 0.6772 - val_recall: 0.7320 - val_auc: 0.8179 - 1s/epoch - 2ms/step
Epoch 14/50
749/749 - 1s - loss: 0.4460 - accuracy: 0.7915 - precision: 0.7285 - recall: 0.7864 - auc: 0.8731 - val_loss: 0.5371 - val_accuracy: 0.7472 - val_precision: 0.6801 - val_recall: 0.7284 - val_auc: 0.8159 - 1s/epoch - 2ms/step
Epoch 15/50
749/749 - 1s - loss: 0.4370 - accuracy: 0.7951 - precision: 0.7336 - recall: 0.7886 - auc: 0.8784 - val_loss: 0.5339 - val_accuracy: 0.7423 - val_precision: 0.6748 - val_recall: 0.7217 - val_auc: 0.8110 - 1s/epoch - 2ms/step
Epoch 16/50
749/749 - 1s - loss: 0.4322 - accuracy: 0.7976 - precision: 0.7338 - recall: 0.7973 - auc: 0.8813 - val_loss: 0.5394 - val_accuracy: 0.7517 - val_precision: 0.6913 - val_recall: 0.7168 - val_auc: 0.8168 - 1s/epoch - 2ms/step
Epoch 17/50
749/749 - 1s - loss: 0.4236 - accuracy: 0.8026 - precision: 0.7401 - recall: 0.8022 - auc: 0.8862 - val_loss: 0.5462 - val_accuracy: 0.7467 - val_precision: 0.6749 - val_recall: 0.7421 - val_auc: 0.8143 - 1s/epoch - 2ms/step
Epoch 18/50
749/749 - 1s - loss: 0.4163 - accuracy: 0.8082 - precision: 0.7464 - recall: 0.8090 - auc: 0.8907 - val_loss: 0.5631 - val_accuracy: 0.7442 - val_precision: 0.6810 - val_recall: 0.7119 - val_auc: 0.8067 - 1s/epoch - 2ms/step
Epoch 19/50
749/749 - 2s - loss: 0.4081 - accuracy: 0.8124 - precision: 0.7524 - recall: 0.8111 - auc: 0.8952 - val_loss: 0.5986 - val_accuracy: 0.7371 - val_precision: 0.6553 - val_recall: 0.7619 - val_auc: 0.8120 - 2s/epoch - 3ms/step
Epoch 20/50
749/749 - 2s - loss: 0.4022 - accuracy: 0.8146 - precision: 0.7520 - recall: 0.8197 - auc: 0.8981 - val_loss: 0.6122 - val_accuracy: 0.7352 - val_precision: 0.6635 - val_recall: 0.7235 - val_auc: 0.8024 - 2s/epoch - 2ms/step
Epoch 21/50
749/749 - 1s - loss: 0.3944 - accuracy: 0.8179 - precision: 0.7580 - recall: 0.8191 - auc: 0.9021 - val_loss: 0.5927 - val_accuracy: 0.7373 - val_precision: 0.6659 - val_recall: 0.7259 - val_auc: 0.8031 - 1s/epoch - 2ms/step
Epoch 22/50
749/749 - 1s - loss: 0.3879 - accuracy: 0.8202 - precision: 0.7582 - recall: 0.8269 - auc: 0.9053 - val_loss: 0.6061 - val_accuracy: 0.7377 - val_precision: 0.6719 - val_recall: 0.7089 - val_auc: 0.7998 - 1s/epoch - 2ms/step
Epoch 23/50
749/749 - 1s - loss: 0.3816 - accuracy: 0.8246 - precision: 0.7617 - recall: 0.8352 - auc: 0.9086 - val_loss: 0.5976 - val_accuracy: 0.7230 - val_precision: 0.6483 - val_recall: 0.7150 - val_auc: 0.7894 - 1s/epoch - 2ms/step
Epoch 24/50
749/749 - 1s - loss: 0.3694 - accuracy: 0.8307 - precision: 0.7692 - recall: 0.8408 - auc: 0.9144 - val_loss: 0.6403 - val_accuracy: 0.7205 - val_precision: 0.6382 - val_recall: 0.7409 - val_auc: 0.7900 - 1s/epoch - 2ms/step
Epoch 25/50
749/749 - 1s - loss: 0.3690 - accuracy: 0.8309 - precision: 0.7700 - recall: 0.8401 - auc: 0.9154 - val_loss: 0.6345 - val_accuracy: 0.7243 - val_precision: 0.6437 - val_recall: 0.7393 - val_auc: 0.7882 - 1s/epoch - 2ms/step
Epoch 26/50
749/749 - 1s - loss: 0.3586 - accuracy: 0.8357 - precision: 0.7736 - recall: 0.8494 - auc: 0.9195 - val_loss: 0.6612 - val_accuracy: 0.7267 - val_precision: 0.6478 - val_recall: 0.7360 - val_auc: 0.7881 - 1s/epoch - 2ms/step
Epoch 27/50
749/749 - 2s - loss: 0.3494 - accuracy: 0.8395 - precision: 0.7781 - recall: 0.8533 - auc: 0.9240 - val_loss: 0.6771 - val_accuracy: 0.7278 - val_precision: 0.6686 - val_recall: 0.6714 - val_auc: 0.7774 - 2s/epoch - 3ms/step

Resultados

plot_metrics(history)

train_pred = rna.predict(X_train)
test_pred = rna.predict(X_test)

print('Número de exemplos positivos do conjunto de teste =', len(y_test[y_test > 0.9]))
results = rna.evaluate(X_test, y_test, verbose=0)

for name, value in zip(rna.metrics_names, results):
    print(f'{name}: {value:.4f}')

749/749 [==============================] - 1s 1ms/step
250/250 [==============================] - 0s 1ms/step
Número de exemplos positivos do conjunto de teste = 3284
loss: 0.5038
accuracy: 0.7597
precision: 0.7151
recall: 0.6918
auc: 0.8282

precision = results[2]
recall = results[3]
F1 = 2*precision*recall/(precision + recall)
print(f'Pontuação F1 = {F1:.4f}')

Pontuação F1 = 0.7033

ConfusionMatrixDisplay.from_predictions(y_test, np.round(test_pred),
                                        normalize='true', cmap='Blues',
                                        values_format='.3f')
plt.show()

from sklearn.metrics import balanced_accuracy_score

balanced_accuracy_score(y_test, np.round(test_pred))

0.7495404835008607

fp_train, tp_train, _ = roc_curve(y_train, train_pred)
fp_test, tp_test, _ = roc_curve(y_test, test_pred)
plt.figure(figsize=(8, 6))
plt.plot(100*fp_train, 100*tp_train, 'b', label='Dados treinamento')
plt.plot(100*fp_test, 100*tp_test, 'r', label='Dados teste')
plt.xlabel('Positivos falsos [%]')
plt.ylabel('Positivos verdadeiros [%]')
plt.xlim([0,100])
plt.ylim([0,100])
plt.grid(True)
plt.legend()
plt.show()

custo_e_metricas_train = rna.evaluate(X_train, y_train)
custo_e_metricas_test = rna.evaluate(X_test, y_test)

749/749 [==============================] - 1s 1ms/step - loss: 0.4650 - accuracy: 0.7830 - precision: 0.7477 - recall: 0.7135 - auc: 0.8567
250/250 [==============================] - 0s 1ms/step - loss: 0.5038 - accuracy: 0.7597 - precision: 0.7151 - recall: 0.6918 - auc: 0.8282

Criação e treinamento da RNA Sequencial

neg, pos = np.bincount(y_train)
total = neg + pos
print(f'Exemplos:\n Total: {total}\n Positivos: {pos} ({100*pos/total:.2f}% do total)')

# Cálculo dos pesos das duas classe
weight_for_0 = (1 / neg)*(total)/2.0
weight_for_1 = (1 / pos)*(total)/2.0

# Dicionário de pesos das classes para treinamento
class_weight = {0: weight_for_0, 1: weight_for_1}
print('Peso da classe 0: {:.2f}'.format(weight_for_0))
print('Peso da classe 1: {:.2f}'.format(weight_for_1))

Exemplos:
 Total: 23937
 Positivos: 9852 (41.16% do total)
Peso da classe 0: 0.85
Peso da classe 1: 1.21

rna = Sequential()
rna.add(Dense(units=128, activation='relu', input_shape=X_train.shape[1:]))
rna.add(Dense(units=128, activation='relu'))
rna.add(Dense(units=32, activation='relu'))
rna.add(Dense(units=1, activation='sigmoid'))
rna.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 dense_12 (Dense)            (None, 128)               3328

 dense_13 (Dense)            (None, 128)               16512

 dense_14 (Dense)            (None, 32)                4128

 dense_15 (Dense)            (None, 1)                 33

=================================================================
Total params: 24,001
Trainable params: 24,001
Non-trainable params: 0
_________________________________________________________________

keras.utils.plot_model(rna, show_shapes=True)

from tensorflow.keras.callbacks import EarlyStopping

# Define métricas
METRICS = [keras.metrics.BinaryAccuracy(name='accuracy'),
           keras.metrics.Precision(name='precision'),
           keras.metrics.Recall(name='recall'),
           keras.metrics.AUC(name='auc')]

call_es = EarlyStopping(monitor='val_loss', patience=20,
                        restore_best_weights=True)

adam = Adam(learning_rate=0.001)
rna.compile(optimizer=adam, loss='binary_crossentropy',
            metrics=METRICS)

history = rna.fit(X_train, y_train, epochs=50,
                  class_weight=class_weight,
                  verbose=2, batch_size=32,
                  validation_data=(X_test, y_test),
                  callbacks=[call_es])

Epoch 1/50
749/749 - 3s - loss: 0.5337 - accuracy: 0.7377 - precision: 0.6618 - recall: 0.7417 - auc: 0.8104 - val_loss: 0.5140 - val_accuracy: 0.7481 - val_precision: 0.6783 - val_recall: 0.7378 - val_auc: 0.8229 - 3s/epoch - 4ms/step
Epoch 2/50
749/749 - 1s - loss: 0.5105 - accuracy: 0.7567 - precision: 0.6893 - recall: 0.7445 - auc: 0.8287 - val_loss: 0.5029 - val_accuracy: 0.7579 - val_precision: 0.7042 - val_recall: 0.7098 - val_auc: 0.8295 - 1s/epoch - 2ms/step
Epoch 3/50
749/749 - 1s - loss: 0.5022 - accuracy: 0.7598 - precision: 0.6945 - recall: 0.7434 - auc: 0.8346 - val_loss: 0.5179 - val_accuracy: 0.7480 - val_precision: 0.6719 - val_recall: 0.7576 - val_auc: 0.8260 - 1s/epoch - 2ms/step
Epoch 4/50
749/749 - 1s - loss: 0.4969 - accuracy: 0.7619 - precision: 0.6976 - recall: 0.7441 - auc: 0.8384 - val_loss: 0.5003 - val_accuracy: 0.7615 - val_precision: 0.7023 - val_recall: 0.7299 - val_auc: 0.8318 - 1s/epoch - 2ms/step
Epoch 5/50
749/749 - 1s - loss: 0.4918 - accuracy: 0.7656 - precision: 0.7018 - recall: 0.7483 - auc: 0.8421 - val_loss: 0.5060 - val_accuracy: 0.7582 - val_precision: 0.6911 - val_recall: 0.7460 - val_auc: 0.8318 - 1s/epoch - 2ms/step
Epoch 6/50
749/749 - 1s - loss: 0.4891 - accuracy: 0.7686 - precision: 0.7056 - recall: 0.7514 - auc: 0.8442 - val_loss: 0.5045 - val_accuracy: 0.7570 - val_precision: 0.6968 - val_recall: 0.7250 - val_auc: 0.8294 - 1s/epoch - 2ms/step
Epoch 7/50
749/749 - 1s - loss: 0.4821 - accuracy: 0.7728 - precision: 0.7125 - recall: 0.7512 - auc: 0.8491 - val_loss: 0.5081 - val_accuracy: 0.7551 - val_precision: 0.6890 - val_recall: 0.7381 - val_auc: 0.8293 - 1s/epoch - 2ms/step
Epoch 8/50
749/749 - 1s - loss: 0.4787 - accuracy: 0.7734 - precision: 0.7124 - recall: 0.7540 - auc: 0.8513 - val_loss: 0.5085 - val_accuracy: 0.7565 - val_precision: 0.6955 - val_recall: 0.7262 - val_auc: 0.8280 - 1s/epoch - 2ms/step
Epoch 9/50
749/749 - 2s - loss: 0.4731 - accuracy: 0.7772 - precision: 0.7173 - recall: 0.7571 - auc: 0.8551 - val_loss: 0.5122 - val_accuracy: 0.7510 - val_precision: 0.6768 - val_recall: 0.7558 - val_auc: 0.8295 - 2s/epoch - 2ms/step
Epoch 10/50
749/749 - 2s - loss: 0.4682 - accuracy: 0.7795 - precision: 0.7188 - recall: 0.7625 - auc: 0.8585 - val_loss: 0.5319 - val_accuracy: 0.7495 - val_precision: 0.6793 - val_recall: 0.7412 - val_auc: 0.8285 - 2s/epoch - 2ms/step
Epoch 11/50
749/749 - 1s - loss: 0.4635 - accuracy: 0.7824 - precision: 0.7250 - recall: 0.7594 - auc: 0.8616 - val_loss: 0.5161 - val_accuracy: 0.7512 - val_precision: 0.6782 - val_recall: 0.7527 - val_auc: 0.8308 - 1s/epoch - 2ms/step
Epoch 12/50
749/749 - 1s - loss: 0.4569 - accuracy: 0.7836 - precision: 0.7246 - recall: 0.7650 - auc: 0.8656 - val_loss: 0.5176 - val_accuracy: 0.7530 - val_precision: 0.6871 - val_recall: 0.7342 - val_auc: 0.8273 - 1s/epoch - 2ms/step
Epoch 13/50
749/749 - 1s - loss: 0.4532 - accuracy: 0.7864 - precision: 0.7276 - recall: 0.7690 - auc: 0.8679 - val_loss: 0.5247 - val_accuracy: 0.7555 - val_precision: 0.6889 - val_recall: 0.7403 - val_auc: 0.8276 - 1s/epoch - 2ms/step
Epoch 14/50
749/749 - 1s - loss: 0.4460 - accuracy: 0.7908 - precision: 0.7335 - recall: 0.7723 - auc: 0.8726 - val_loss: 0.5253 - val_accuracy: 0.7502 - val_precision: 0.6770 - val_recall: 0.7518 - val_auc: 0.8271 - 1s/epoch - 2ms/step
Epoch 15/50
749/749 - 1s - loss: 0.4392 - accuracy: 0.7950 - precision: 0.7382 - recall: 0.7777 - auc: 0.8766 - val_loss: 0.5582 - val_accuracy: 0.7346 - val_precision: 0.6474 - val_recall: 0.7795 - val_auc: 0.8231 - 1s/epoch - 2ms/step
Epoch 16/50
749/749 - 1s - loss: 0.4348 - accuracy: 0.7992 - precision: 0.7432 - recall: 0.7824 - auc: 0.8791 - val_loss: 0.5233 - val_accuracy: 0.7493 - val_precision: 0.7014 - val_recall: 0.6809 - val_auc: 0.8167 - 1s/epoch - 2ms/step
Epoch 17/50
749/749 - 1s - loss: 0.4272 - accuracy: 0.8012 - precision: 0.7466 - recall: 0.7825 - auc: 0.8835 - val_loss: 0.5361 - val_accuracy: 0.7486 - val_precision: 0.6854 - val_recall: 0.7192 - val_auc: 0.8241 - 1s/epoch - 2ms/step
Epoch 18/50
749/749 - 2s - loss: 0.4214 - accuracy: 0.8045 - precision: 0.7494 - recall: 0.7887 - auc: 0.8868 - val_loss: 0.5432 - val_accuracy: 0.7515 - val_precision: 0.6916 - val_recall: 0.7150 - val_auc: 0.8208 - 2s/epoch - 3ms/step
Epoch 19/50
749/749 - 2s - loss: 0.4149 - accuracy: 0.8098 - precision: 0.7557 - recall: 0.7946 - auc: 0.8907 - val_loss: 0.5579 - val_accuracy: 0.7366 - val_precision: 0.6637 - val_recall: 0.7296 - val_auc: 0.8106 - 2s/epoch - 2ms/step
Epoch 20/50
749/749 - 1s - loss: 0.4107 - accuracy: 0.8110 - precision: 0.7563 - recall: 0.7981 - auc: 0.8932 - val_loss: 0.5552 - val_accuracy: 0.7338 - val_precision: 0.6663 - val_recall: 0.7077 - val_auc: 0.8047 - 1s/epoch - 2ms/step
Epoch 21/50
749/749 - 1s - loss: 0.3979 - accuracy: 0.8180 - precision: 0.7659 - recall: 0.8035 - auc: 0.9006 - val_loss: 0.5671 - val_accuracy: 0.7391 - val_precision: 0.6716 - val_recall: 0.7162 - val_auc: 0.8105 - 1s/epoch - 2ms/step
Epoch 22/50
749/749 - 1s - loss: 0.3959 - accuracy: 0.8178 - precision: 0.7665 - recall: 0.8015 - auc: 0.9012 - val_loss: 0.5803 - val_accuracy: 0.7378 - val_precision: 0.6668 - val_recall: 0.7256 - val_auc: 0.8071 - 1s/epoch - 2ms/step
Epoch 23/50
749/749 - 1s - loss: 0.3872 - accuracy: 0.8230 - precision: 0.7707 - recall: 0.8111 - auc: 0.9058 - val_loss: 0.5791 - val_accuracy: 0.7413 - val_precision: 0.6820 - val_recall: 0.6961 - val_auc: 0.8056 - 1s/epoch - 2ms/step
Epoch 24/50
749/749 - 1s - loss: 0.3807 - accuracy: 0.8264 - precision: 0.7737 - recall: 0.8173 - auc: 0.9092 - val_loss: 0.5840 - val_accuracy: 0.7373 - val_precision: 0.6729 - val_recall: 0.7040 - val_auc: 0.8064 - 1s/epoch - 2ms/step

Resultados

plot_metrics(history)

train_pred = rna.predict(X_train)
test_pred = rna.predict(X_test)

print('Número de exemplos positivos do conjunto de teste =', len(y_test[y_test > 0.9]))
results = rna.evaluate(X_test, y_test, verbose=0)

for name, value in zip(rna.metrics_names, results):
    print(f'{name}: {value:.4f}')

749/749 [==============================] - 1s 939us/step
250/250 [==============================] - 0s 939us/step
Número de exemplos positivos do conjunto de teste = 3284
loss: 0.5003
accuracy: 0.7615
precision: 0.7023
recall: 0.7299
auc: 0.8318

precision = results[2]
recall = results[3]
F1 = 2*precision*recall/(precision + recall)
print(f'Pontuação F1 = {F1:.4f}')

Pontuação F1 = 0.7158

ConfusionMatrixDisplay.from_predictions(y_test, np.round(test_pred),
                                        normalize='true', cmap='Blues',
                                        values_format='.3f')
plt.show()

from sklearn.metrics import balanced_accuracy_score

balanced_accuracy_score(y_test, np.round(test_pred))

0.7567510659355912

fp_train, tp_train, _ = roc_curve(y_train, train_pred)
fp_test, tp_test, _ = roc_curve(y_test, test_pred)
plt.figure(figsize=(8, 6))
plt.plot(100*fp_train, 100*tp_train, 'b', label='Dados treinamento')
plt.plot(100*fp_test, 100*tp_test, 'r', label='Dados teste')
plt.xlabel('Positivos falsos [%]')
plt.ylabel('Positivos verdadeiros [%]')
plt.xlim([0,100])
plt.ylim([0,100])
plt.grid(True)
plt.legend()
plt.show()

custo_e_metricas_train = rna.evaluate(X_train, y_train)
custo_e_metricas_test = rna.evaluate(X_test, y_test)

749/749 [==============================] - 1s 2ms/step - loss: 0.4778 - accuracy: 0.7733 - precision: 0.7230 - recall: 0.7282 - auc: 0.8476
250/250 [==============================] - 0s 2ms/step - loss: 0.5003 - accuracy: 0.7615 - precision: 0.7023 - recall: 0.7299 - auc: 0.8318