Introdução

Ideia Geral

Utilizando os dados de câncer da FOSP, somente do tipo colorretal, serão utilizados quatro modelos de machine learning diferentes, com o intuito de testar diferentes tipos de algoritmo na classificação do óbito por qualquer razão.

O label é 0 se o paciente está vivo e 1 se ele morreu por qualquer razão.

Modelos de ML

Foram escolhidos os modelos Naive Bayes, que utiliza o Teorema de Bayes para realizar as previsões, Random Forest, XGBoost e LightGBM, que utilizam os conceitos de árvores de decisão, além de bagging e boosting. Além disso, será testado um modelo de votação com os melhores classificadores obtidos, visando obter um algoritmo ainda mais acertivo.

Validação dos modelos

Para validar os modelos treinados foi utilizada primeiramente a matriz de confusão, sendo possível avaliar os acertos em ambas as classes. Para entender de houve overfitting nos modelos, foi utilizada a curva ROC para os conjuntos de treino e teste, comparando a métrica AUC entre ambos os conjuntos.

Por fim, os modelos Random Forest, XGBoost e LightGBM oferecem a possibilidade de saber quais foram as features mais importantes, ou seja, que mais influenciam na previsão das classes. Assim, foram mostradas duas maneiras diferentes de analisar a importância das variáveis de entrada, uma usando a própria função dos modelos e outra usando a biblioteca SHAP, que mostra a influência das features em ambas as classes.

# Leitura dos dados
df = read_csv('/content/drive/MyDrive/Trabalho/Cancer/Datasets/colorretal_labels.csv')
df.head(3)

(31916, 37)

	ESCOLARI	IDADE	SEXO	IBGE	CATEATEND	DIAGPREV	EC	ECGRUP	TRATHOSP	NENHUM	...	IBGEATEN	ULTICONS	ULTIDIAG	ULTITRAT	obito_geral	obito_cancer	vivo_ano1	vivo_ano3	vivo_ano5	ESCOLARI_2
0	4	19	2	3538709	9	2	IV	IV	I	0	...	3538709	4985	4985	4951	0	0	1	1	1	4.0
1	9	19	1	3537107	2	2	IIIA	III	I	0	...	3509502	2680	2744	2674	1	1	1	1	1	4.0
2	4	19	1	3516200	9	2	IIB	II	F	0	...	3516200	4725	4734	4719	0	0	1	1	1	4.0

3 rows × 37 columns

# Valores faltantes
df.isna().sum().sort_values(ascending=False).head(6)

ESCOLARI     0
CONSDIAG     0
DIAGTRAT     0
ANODIAG      0
FAIXAETAR    0
DRS          0
dtype: int64

# Correlação com a saída
corr_matrix = df.corr()
abs(corr_matrix['obito_geral']).sort_values(ascending = False).head(20)

The default value of numeric_only in DataFrame.corr is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning.

obito_geral     1.000000
ULTINFO         0.867403
obito_cancer    0.788496
ULTIDIAG        0.380830
ULTITRAT        0.377590
ULTICONS        0.375966
vivo_ano3       0.370056
vivo_ano5       0.344902
ANODIAG         0.266910
vivo_ano1       0.261431
CATEATEND       0.210779
RECNENHUM       0.189673
CIRURGIA        0.142206
IDADE           0.113836
ESCOLARI_2      0.098022
DIAGPREV        0.097647
IBGEATEN        0.047572
SEXO            0.047430
QUIMIO          0.046282
RADIO           0.045016
Name: obito_geral, dtype: float64

# Quantidade de pacientes em cada classe da saída
df.obito_geral.value_counts()

1    16897
0    15019
Name: obito_geral, dtype: int64

Análise - Óbito geral

Pré-processamento

Como o dataset já foi limpo anteriormente, aqui na etapa de pré-processamento serão realizadas a divisão dos dados em treino e teste, a codificação das colunas textuais para colunas numéricas e a normalização dos dados. Com isso, temos os dados prontos para o treinamento dos modelos de machine learning e consequentemente sua validação.

Neste primeiro momento, serão definidas as colunas que não serão utilizadas como features, assim, foi escolhido manter a coluna IDADE, então a coluna FAIXAETAR será retirada. O mesmo ocorre com a coluna EC, retirando a coluna ECGRUP. Por fim, as outras colunas contidas na list_drop são possíveis saídas para os modelos, mas estamos interessados somente no óbito geral, por isso só ela será mantida como label e as outras serão retiradas.

list_drop = ['FAIXAETAR', 'ULTICONS', 'ULTIDIAG', 'ULTITRAT', 'vivo_ano1',
             'vivo_ano3', 'vivo_ano5', 'ULTINFO', 'obito_cancer', 'ECGRUP', 'ESCOLARI']

lb = 'obito_geral'

Uma função foi criada para realizar o pré-processamento inteiro, chamada preprocessing, internamente ela utiliza outras funções criadas que são: get_train_test (divide os dados em treino e teste), train_preprocessing (prepara os dados de treino) e test_preprocessing (prepara os dados de teste).

Mais detalhes em funções.

X_train, X_test, y_train, y_test, feat_cols = preprocessing(df, list_drop, lb,
                                                            random_state=seed,
                                                            balance_data=False,
                                                            encoder_type='LabelEncoder',
                                                            norm_name='StandardScaler')

X_train = (23937, 25), X_test = (7979, 25)
y_train = (23937,), y_test = (7979,)

y_train.value_counts(normalize=True)

1    0.529431
0    0.470569
Name: obito_geral, dtype: float64

y_test.value_counts(normalize=True)

1    0.52939
0    0.47061
Name: obito_geral, dtype: float64

Treinamento e validação dos modelos de machine learning

Depois das etapas de preparação, os dados estão prontos para serem utilizados nos modelos escolhidos.

Naive Bayes

# Criação e treinamento do modelo Naive Bayes
nb = GaussianNB()
nb.fit(X_train, y_train)

GaussianNB()

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# Matriz de confusão
plot_confusion_matrix(nb, X_test, y_test)

              precision    recall  f1-score   support

           0      0.471     1.000     0.640      3755
           1      1.000     0.001     0.002      4224

    accuracy                          0.471      7979
   macro avg      0.735     0.500     0.321      7979
weighted avg      0.751     0.471     0.302      7979

Claramente percebe-se que o modelo previu quase todos os dados como sendo da classe 0, portanto não teve um aprendizado satisfatório.

Na matriz de confusão, buscamos uma diagonal principal equilibrada e com a maior acertividade possível.

# Curva ROC
plot_roc_curve(nb, X_train, X_test, y_train, y_test)

Pelas curvas ROC, pode-se dizer que não há overfitting, mas o modelo é ruim para a previsão da classe 1, portanto qualquer análise além dessa não possui tanta relevância.

Random Forest

O modelo Random Forest é mais complexo em relação ao Naive Bayes, assim alguns hiperparâmetros serão definidos para obter um modelo base e depois será realizada a busca dos melhores parâmetros utilizando o Optuna.

Os parâmetros definidos para este primeiro modelo serão:

• random_state: para repetibilidade do treinamento do modelo. Será utilizado na busca pelos hiperparâmetros também, sempre como mesmo valor definido na variável seed.

• max_depth: será definido como 8, pois o padrão do modelo é não ter profundidade máxima para as árvores, o que dificulta e faz o treinamento ser muito longo, além da maior chance de overfitting.

• class_weight: usado para definir os pesos de cada classe no treinamento do modelo, muito útil quando temos classes desbalanceadas no conjunto de dados, como neste caso.

# Criação e treinamento do modelo Random Forest
rf = RandomForestClassifier(random_state=seed,
                            max_depth=8,
                            class_weight={0:1.028, 1:1},
                            criterion='entropy')

rf.fit(X_train, y_train)

RandomForestClassifier(class_weight={0: 1.028, 1: 1}, criterion='entropy',
                       max_depth=8, random_state=10)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# Matriz de confusão
plot_confusion_matrix(rf, X_test, y_test)

              precision    recall  f1-score   support

           0      0.738     0.761     0.749      3755
           1      0.781     0.760     0.771      4224

    accuracy                          0.760      7979
   macro avg      0.760     0.760     0.760      7979
weighted avg      0.761     0.760     0.761      7979

A matriz obtida para o modelo Random Forest apresentou diagonal equilibrada em ambas as classes, com 76% de acurácia.

# Pedaço de uma das árvores do modelo Random Forest
show_tree(rf, feat_cols, 2)

# Curva ROC
plot_roc_curve(rf, X_train, X_test, y_train, y_test)

Como a métrica AUC possui valores próximos para o conjunto de treino e de teste, 0,861 e 0,843 respectivamente, pode-se dizer que há apenas um pouco de overfitting, não sendo algo de grande preocupação.

# Importância das features
plot_feat_importances(rf, feat_cols)

As features mais importantes nesta visualização são EC, com uma grande vantagem, ANODIAG, RECNENHUM e CATEATEND.

# Importância das features pelos valores SHAP
plot_shap_values(rf, X_train, feat_cols)

A coluna EC foi a mais importante aqui também, com isso, os valores mais altos desta variável, mostrados em rosa, influenciaram mais o modelo na previsão da classe 1 (óbito por qualquer razão). Já os valores mais baixos desta coluna, em azul, tem mais peso para previsão ser da classe 0. Este comportamento faz sentido, pois quanto mais alto o estágio, maior é a extensão do câncer, assim menor a chance de sobrevivência.

O raciocínio para analisar as outras colunas é o mesmo utilizado para o estadiamento clínico.

Optuna

Para fazer a busca pelos melhores hiperparâmetros, será utilizado a biblioteca Optuna, definindo o intervalo para os parâmetros do modelo a serem buscados.

# Folds com a mesma proporção das classes
skf = StratifiedKFold(10, shuffle=True, random_state=seed)

# Função com o modelos e seus parâmetros, que terá sua métrica maximizada
def objective(trial):

    n_estimators = trial.suggest_int('n_estimators', 50, 250)
    max_depth = trial.suggest_int('max_depth', 3, 18)
    min_samples_split = trial.suggest_int('min_samples_split', 2, 10)
    min_samples_leaf = trial.suggest_int('min_samples_leaf', 1, 7)
    max_samples = trial.suggest_float('max_samples', 0.7, 1.0, step=0.1)
    optimizer = trial.suggest_categorical('criterion', ['gini', 'entropy'])

    cls = RandomForestClassifier(n_estimators=n_estimators,
                                 max_depth=max_depth,
                                 min_samples_split=min_samples_split,
                                 min_samples_leaf=min_samples_leaf,
                                 max_samples=max_samples,
                                 criterion=optimizer,
                                 random_state=seed)

    return cross_val_score(cls, X_train, y_train,
                           cv=skf, scoring='balanced_accuracy').mean()

# Criação do estudo e procura pelos hiperparâmetros
studyRF = optuna.create_study(direction='maximize', sampler=RandomSampler(seed))
studyRF.optimize(objective, n_trials=100)

# Melhor tentativa
studyRF.best_trial

FrozenTrial(number=69, state=TrialState.COMPLETE, values=[0.7659635302997903], datetime_start=datetime.datetime(2023, 4, 11, 23, 41, 4, 440279), datetime_complete=datetime.datetime(2023, 4, 11, 23, 41, 45, 938175), params={'n_estimators': 173, 'max_depth': 17, 'min_samples_split': 8, 'min_samples_leaf': 4, 'max_samples': 0.7999999999999999, 'criterion': 'entropy'}, user_attrs={}, system_attrs={}, intermediate_values={}, distributions={'n_estimators': IntDistribution(high=250, log=False, low=50, step=1), 'max_depth': IntDistribution(high=18, log=False, low=3, step=1), 'min_samples_split': IntDistribution(high=10, log=False, low=2, step=1), 'min_samples_leaf': IntDistribution(high=7, log=False, low=1, step=1), 'max_samples': FloatDistribution(high=1.0, log=False, low=0.7, step=0.1), 'criterion': CategoricalDistribution(choices=('gini', 'entropy'))}, trial_id=69, value=None)

# Melhores parâmetros
studyRF.best_params

{'n_estimators': 173,
 'max_depth': 17,
 'min_samples_split': 8,
 'min_samples_leaf': 4,
 'max_samples': 0.7999999999999999,
 'criterion': 'entropy'}

plot_optimization_history(studyRF).show()

# Modelo com os melhores parâmetros
params = studyRF.best_params
params['random_state'] = seed
params['class_weight'] = {0: 1.11, 1: 1}

rf_optuna = RandomForestClassifier()
rf_optuna.set_params(**params)

rf_optuna.fit(X_train, y_train)

RandomForestClassifier(class_weight={0: 1.11, 1: 1}, criterion='entropy',
                       max_depth=17, max_samples=0.7999999999999999,
                       min_samples_leaf=4, min_samples_split=8,
                       n_estimators=173, random_state=10)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# Matriz de confusão do modelo Random Forest otimizado
plot_confusion_matrix(rf_optuna, X_test, y_test)

              precision    recall  f1-score   support

           0      0.751     0.773     0.762      3755
           1      0.793     0.772     0.782      4224

    accuracy                          0.772      7979
   macro avg      0.772     0.772     0.772      7979
weighted avg      0.773     0.772     0.772      7979

Há uma melhora de acurácia em relação ao primeiro modelo testado, de 76% para 77%.

# Curva ROC do modelo otimizado
plot_roc_curve(rf_optuna, X_train, X_test, y_train, y_test)

A curva ROC mostra que o modelo possui overfitting, pois para o conjunto de treino temos AUC = 0,958 e para o teste AUC = 0,852, essa diferença caracteriza o problema.

# Importância das features pelos valores SHAP
plot_shap_values(rf_optuna, X_train, feat_cols)

No data for colormapping provided via 'c'. Parameters 'vmin', 'vmax' will be ignored

XGBoost

O modelo XGBoost também terá alguns hiperparâmetros definidos para obter um modelo base e depois será realizada a busca dos melhores parâmetros utilizando o Optuna.

Os parâmetros definidos para este primeiro modelo serão:

• random_state: para repetibilidade do treinamento do modelo. Será utilizado na busca pelos hiperparâmetros também, sempre como mesmo valor definido na variável seed.

• max_depth: será definido como 4, pois o padrão do modelo é 3 de profundidade máxima para as árvores, deixando o modelo muito simples.

• scale_pos_weight: usado para definir o peso da classe 1 no treinamento do modelo, pois temos classes desbalanceadas.

# Criação e treinamento do modelo XGBoost
xgb = XGBClassifier(max_depth=4,
                    scale_pos_weight=0.918,
                    random_state=seed)

xgb.fit(X_train, y_train)

XGBClassifier(base_score=None, booster=None, callbacks=None,
              colsample_bylevel=None, colsample_bynode=None,
              colsample_bytree=None, early_stopping_rounds=None,
              enable_categorical=False, eval_metric=None, feature_types=None,
              gamma=None, gpu_id=None, grow_policy=None, importance_type=None,
              interaction_constraints=None, learning_rate=None, max_bin=None,
              max_cat_threshold=None, max_cat_to_onehot=None,
              max_delta_step=None, max_depth=4, max_leaves=None,
              min_child_weight=None, missing=nan, monotone_constraints=None,
              n_estimators=100, n_jobs=None, num_parallel_tree=None,
              predictor=None, random_state=10, ...)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# Matriz de confusão
plot_confusion_matrix(xgb, X_test, y_test)

              precision    recall  f1-score   support

           0      0.755     0.777     0.766      3755
           1      0.797     0.776     0.786      4224

    accuracy                          0.776      7979
   macro avg      0.776     0.776     0.776      7979
weighted avg      0.777     0.776     0.777      7979

A matriz obtida para o modelo XGBoost apresentou diagonal equilibrada em ambas as classes, com 78% de acurácia.

# Curva ROC
plot_roc_curve(xgb, X_train, X_test, y_train, y_test)

Como a métrica AUC possui valores próximos para o conjunto de treino e de teste, 0,887 e 0,854 respectivamente, pode-se dizer que há bem pouco overfitting, não sendo algo de grande preocupação.

# Importância das features
plot_feat_importances(xgb, feat_cols)

As features mais importantes nesta visualização são EC, RECNENHUM, ANODIAG e CATEATEND.

# Importância das features pelo valores SHAP
plot_shap_values(xgb, X_train, feat_cols)

A coluna EC foi a mais importante também, com isso, os valores mais altos desta variável, mostrados em rosa, influenciaram mais o modelo na previsão da classe 1 (óbito por câncer). Já os valores mais baixos desta coluna, em azul, tem mais peso para previsão ser da classe 0. Este comportamento faz sentido, pois quanto mais alto o estágio, maior é a extensão do câncer, assim menor a chance de sobrevivência.

O raciocínio para analisar as outras colunas é o mesmo utilizado para o estadiamento clínico.

Optuna

Para fazer a busca pelos melhores hiperparâmetros, será utilizado a biblioteca Optuna, definindo o intervalo para os parâmetros do modelo a serem buscados.

# Folds com a mesma proporção das classes
skf = StratifiedKFold(10, shuffle=True, random_state=seed)

# Função com o modelos e seus parâmetros, que terá sua métrica maximizada
def objective(trial):

    n_estimators = trial.suggest_int('n_estimators', 50, 250)
    max_depth = trial.suggest_int('max_depth', 3, 18)
    learning_rate = trial.suggest_float('learning_rate', 0.05, 0.2, step=0.05)
    gamma = trial.suggest_float('gamma', 0.0, 0.3, step=0.1)
    min_child_weight = trial.suggest_int('min_child_weight', 1, 7)
    colsample_bytree = trial.suggest_float('colsample_bytree', 0.3, 0.7, step=0.1)

    cls = XGBClassifier(n_estimators=n_estimators,
                        max_depth=max_depth,
                        learning_rate=learning_rate,
                        gamma=gamma,
                        min_child_weight=min_child_weight,
                        colsample_bytree=colsample_bytree,
                        random_state=seed)

    return cross_val_score(cls, X_train, y_train,
                           cv=skf, scoring='balanced_accuracy').mean()

# Criação do estudo e procura pelos hiperparâmetros
studyXGB = optuna.create_study(direction='maximize', sampler=RandomSampler(seed))
studyXGB.optimize(objective, n_trials=100)

# Melhor tentativa
studyXGB.best_trial

# Melhores parâmetros
studyXGB.best_params

{'n_estimators': 170,
 'max_depth': 4,
 'learning_rate': 0.1,
 'gamma': 0.2,
 'min_child_weight': 4,
 'colsample_bytree': 0.7}

plot_optimization_history(studyXGB).show()

# Modelo com os melhores parâmetros
params = studyXGB.best_params
params['random_state'] = seed
params['scale_pos_weight'] = 0.93

xgb_optuna = XGBClassifier()
xgb_optuna.set_params(**params)

xgb_optuna.fit(X_train, y_train)

XGBClassifier(base_score=None, booster=None, callbacks=None,
              colsample_bylevel=None, colsample_bynode=None,
              colsample_bytree=0.7, early_stopping_rounds=None,
              enable_categorical=False, eval_metric=None, feature_types=None,
              gamma=0.2, gpu_id=None, grow_policy=None, importance_type=None,
              interaction_constraints=None, learning_rate=0.1, max_bin=None,
              max_cat_threshold=None, max_cat_to_onehot=None,
              max_delta_step=None, max_depth=4, max_leaves=None,
              min_child_weight=4, missing=nan, monotone_constraints=None,
              n_estimators=170, n_jobs=None, num_parallel_tree=None,
              predictor=None, random_state=10, ...)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# Matriz de confusão do modelo XGBoost otimizado
plot_confusion_matrix(xgb_optuna, X_test, y_test)

              precision    recall  f1-score   support

           0      0.756     0.777     0.766      3755
           1      0.797     0.777     0.786      4224

    accuracy                          0.777      7979
   macro avg      0.776     0.777     0.776      7979
weighted avg      0.777     0.777     0.777      7979

Há uma melhora de acurácia em relação ao primeiro modelo testado, de 77% para 78%.

# Curva ROC do modelo otimizado
plot_roc_curve(xgb_optuna, X_train, X_test, y_train, y_test)

A curva ROC mostra que o modelo possui um pouco de overfitting, pois para o conjunto de treino temos AUC = 0,873 e para o teste AUC = 0,857, essa diferença caracteriza o problema.

# Importância das features pelo valores SHAP
plot_shap_values(xgb_optuna, X_train, feat_cols)

ROCs

# Treino
roc_together(X_train, y_train, nb, rf_optuna, xgb_optuna)

# Teste
roc_together(X_test, y_test, nb, rf_optuna, xgb_optuna)

RN

Importação das bibliotecas e funções

import tensorflow as tf
from tensorflow import keras

import matplotlib.pyplot as plt

from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Dense, Dropout, Add, Input, Activation
from tensorflow.keras.optimizers import Adam

from tensorflow.keras.regularizers import l2
from tensorflow.keras.callbacks import EarlyStopping

# Definição de cores para gráficos
colors = plt.rcParams['axes.prop_cycle'].by_key()['color']

def plot_metrics(history):
    """Plot metrics after training the RNA.

    :param history: RNA training history.

    :return: no value
    :rtype: none
    """
    metrics = ['loss', 'accuracy', 'precision', 'recall']
    plt.figure(figsize=(12,8))
    for n, metric in enumerate(metrics):
        name = metric.replace("_"," ").capitalize()
        plt.subplot(2, 2, n + 1)
        plt.plot(history.epoch, history.history[metric], color=colors[0], label='Train')
        plt.plot(history.epoch, history.history['val_'+ metric],
                 color=colors[0], linestyle="--", label='Val')
        plt.xlabel('Epoch')
        plt.ylabel(name)
        if metric == 'loss':
            plt.ylim([0, plt.ylim()[1]])
        if metric == 'accuracy':
            plt.ylim([0.7, 1])
        else:
            plt.ylim([0, 1])
        plt.legend()

Criação e treinamento da RNA Complexa

neg, pos = np.bincount(y_train)
total = neg + pos
print(f'Exemplos:\n Total: {total}\n Positivos: {pos} ({100*pos/total:.2f}% do total)')

# Cálculo dos pesos das duas classe
weight_for_0 = (1 / neg)*(total)/2.0
weight_for_1 = (1 / pos)*(total)/2.0

# Dicionário de pesos das classes para treinamento
class_weight = {0: weight_for_0, 1: weight_for_1}
print('Peso da classe 0: {:.2f}'.format(weight_for_0))
print('Peso da classe 1: {:.2f}'.format(weight_for_1))

Exemplos:
 Total: 23937
 Positivos: 12673 (52.94% do total)
Peso da classe 0: 1.06
Peso da classe 1: 0.94

input_shape = X_train.shape[1:]
input_features = Input(shape=input_shape, name='input_features')

x1 = Dense(128, activation='tanh', kernel_regularizer=l2())(input_features)
x2 = Dense(128, activation='selu', kernel_regularizer=l2())(input_features)
x3 = Dense(128, activation='sigmoid', kernel_regularizer=l2())(input_features)

from tensorflow.keras.layers import Concatenate
x_concat = Concatenate()([x1, x2, x3, input_features])

x4 = Dense(32, activation='relu', kernel_regularizer=l2())(x_concat)
out = Dense(1, activation='sigmoid', name='out_dense')(x4)

model = keras.Model(inputs=[input_features],
                    outputs=[out])

model.summary()

Model: "model"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to
==================================================================================================
 input_features (InputLayer)    [(None, 25)]         0           []

 dense (Dense)                  (None, 128)          3328        ['input_features[0][0]']

 dense_1 (Dense)                (None, 128)          3328        ['input_features[0][0]']

 dense_2 (Dense)                (None, 128)          3328        ['input_features[0][0]']

 concatenate (Concatenate)      (None, 409)          0           ['dense[0][0]',
                                                                  'dense_1[0][0]',
                                                                  'dense_2[0][0]',
                                                                  'input_features[0][0]']

 dense_3 (Dense)                (None, 32)           13120       ['concatenate[0][0]']

 out_dense (Dense)              (None, 1)            33          ['dense_3[0][0]']

==================================================================================================
Total params: 23,137
Trainable params: 23,137
Non-trainable params: 0
__________________________________________________________________________________________________

keras.utils.plot_model(model, show_shapes=True)

from tensorflow.keras.callbacks import EarlyStopping

# Define métricas
METRICS = [keras.metrics.BinaryAccuracy(name='accuracy'),
           keras.metrics.Precision(name='precision'),
           keras.metrics.Recall(name='recall'),
           keras.metrics.AUC(name='auc')]

call_es = EarlyStopping(monitor='val_loss', patience=20, restore_best_weights=True)

adam = Adam(learning_rate=0.001)
model.compile(optimizer=adam, loss='binary_crossentropy',
              metrics=METRICS)

history = model.fit(X_train, y_train, epochs=50,
                    class_weight=class_weight,
                    verbose=2, batch_size=32,
                    validation_data=(X_test, y_test),
                    callbacks=[call_es])

Epoch 1/50
749/749 - 5s - loss: 0.8399 - accuracy: 0.7260 - precision: 0.7473 - recall: 0.7290 - auc: 0.7992 - val_loss: 0.5664 - val_accuracy: 0.7364 - val_precision: 0.7830 - val_recall: 0.6946 - val_auc: 0.8159 - 5s/epoch - 6ms/step
Epoch 2/50
749/749 - 2s - loss: 0.5594 - accuracy: 0.7307 - precision: 0.7530 - recall: 0.7310 - auc: 0.8058 - val_loss: 0.5441 - val_accuracy: 0.7394 - val_precision: 0.7497 - val_recall: 0.7623 - val_auc: 0.8163 - 2s/epoch - 3ms/step
Epoch 3/50
749/749 - 2s - loss: 0.5519 - accuracy: 0.7322 - precision: 0.7556 - recall: 0.7305 - auc: 0.8074 - val_loss: 0.5349 - val_accuracy: 0.7436 - val_precision: 0.7579 - val_recall: 0.7576 - val_auc: 0.8213 - 2s/epoch - 3ms/step
Epoch 4/50
749/749 - 3s - loss: 0.5471 - accuracy: 0.7378 - precision: 0.7616 - recall: 0.7349 - auc: 0.8112 - val_loss: 0.5347 - val_accuracy: 0.7435 - val_precision: 0.7333 - val_recall: 0.8099 - val_auc: 0.8237 - 3s/epoch - 4ms/step
Epoch 5/50
749/749 - 3s - loss: 0.5460 - accuracy: 0.7351 - precision: 0.7583 - recall: 0.7334 - auc: 0.8128 - val_loss: 0.5377 - val_accuracy: 0.7442 - val_precision: 0.7869 - val_recall: 0.7088 - val_auc: 0.8267 - 3s/epoch - 4ms/step
Epoch 6/50
749/749 - 2s - loss: 0.5437 - accuracy: 0.7386 - precision: 0.7623 - recall: 0.7356 - auc: 0.8144 - val_loss: 0.5289 - val_accuracy: 0.7483 - val_precision: 0.7439 - val_recall: 0.8000 - val_auc: 0.8285 - 2s/epoch - 3ms/step
Epoch 7/50
749/749 - 2s - loss: 0.5401 - accuracy: 0.7398 - precision: 0.7627 - recall: 0.7382 - auc: 0.8172 - val_loss: 0.5307 - val_accuracy: 0.7503 - val_precision: 0.7791 - val_recall: 0.7375 - val_auc: 0.8294 - 2s/epoch - 3ms/step
Epoch 8/50
749/749 - 2s - loss: 0.5407 - accuracy: 0.7411 - precision: 0.7631 - recall: 0.7410 - auc: 0.8172 - val_loss: 0.5283 - val_accuracy: 0.7468 - val_precision: 0.7540 - val_recall: 0.7744 - val_auc: 0.8267 - 2s/epoch - 3ms/step
Epoch 9/50
749/749 - 2s - loss: 0.5381 - accuracy: 0.7420 - precision: 0.7652 - recall: 0.7397 - auc: 0.8184 - val_loss: 0.5557 - val_accuracy: 0.7275 - val_precision: 0.8317 - val_recall: 0.6084 - val_auc: 0.8304 - 2s/epoch - 3ms/step
Epoch 10/50
749/749 - 3s - loss: 0.5381 - accuracy: 0.7425 - precision: 0.7645 - recall: 0.7424 - auc: 0.8187 - val_loss: 0.5331 - val_accuracy: 0.7501 - val_precision: 0.7704 - val_recall: 0.7521 - val_auc: 0.8252 - 3s/epoch - 4ms/step
Epoch 11/50
749/749 - 3s - loss: 0.5358 - accuracy: 0.7430 - precision: 0.7666 - recall: 0.7398 - auc: 0.8204 - val_loss: 0.5263 - val_accuracy: 0.7500 - val_precision: 0.7359 - val_recall: 0.8232 - val_auc: 0.8316 - 3s/epoch - 5ms/step
Epoch 12/50
749/749 - 3s - loss: 0.5365 - accuracy: 0.7424 - precision: 0.7638 - recall: 0.7432 - auc: 0.8206 - val_loss: 0.5251 - val_accuracy: 0.7511 - val_precision: 0.7393 - val_recall: 0.8184 - val_auc: 0.8309 - 3s/epoch - 4ms/step
Epoch 13/50
749/749 - 3s - loss: 0.5336 - accuracy: 0.7469 - precision: 0.7677 - recall: 0.7483 - auc: 0.8222 - val_loss: 0.5182 - val_accuracy: 0.7554 - val_precision: 0.7615 - val_recall: 0.7831 - val_auc: 0.8330 - 3s/epoch - 5ms/step
Epoch 14/50
749/749 - 2s - loss: 0.5323 - accuracy: 0.7493 - precision: 0.7722 - recall: 0.7468 - auc: 0.8230 - val_loss: 0.5272 - val_accuracy: 0.7460 - val_precision: 0.7231 - val_recall: 0.8428 - val_auc: 0.8330 - 2s/epoch - 3ms/step
Epoch 15/50
749/749 - 3s - loss: 0.5303 - accuracy: 0.7483 - precision: 0.7704 - recall: 0.7472 - auc: 0.8239 - val_loss: 0.5222 - val_accuracy: 0.7545 - val_precision: 0.7605 - val_recall: 0.7827 - val_auc: 0.8306 - 3s/epoch - 4ms/step
Epoch 16/50
749/749 - 3s - loss: 0.5301 - accuracy: 0.7470 - precision: 0.7691 - recall: 0.7461 - auc: 0.8244 - val_loss: 0.5179 - val_accuracy: 0.7525 - val_precision: 0.7552 - val_recall: 0.7879 - val_auc: 0.8322 - 3s/epoch - 4ms/step
Epoch 17/50
749/749 - 2s - loss: 0.5286 - accuracy: 0.7496 - precision: 0.7708 - recall: 0.7499 - auc: 0.8252 - val_loss: 0.5146 - val_accuracy: 0.7561 - val_precision: 0.7746 - val_recall: 0.7607 - val_auc: 0.8362 - 2s/epoch - 3ms/step
Epoch 18/50
749/749 - 2s - loss: 0.5277 - accuracy: 0.7502 - precision: 0.7718 - recall: 0.7499 - auc: 0.8258 - val_loss: 0.5153 - val_accuracy: 0.7581 - val_precision: 0.7719 - val_recall: 0.7708 - val_auc: 0.8356 - 2s/epoch - 3ms/step
Epoch 19/50
749/749 - 2s - loss: 0.5274 - accuracy: 0.7517 - precision: 0.7743 - recall: 0.7494 - auc: 0.8265 - val_loss: 0.5195 - val_accuracy: 0.7557 - val_precision: 0.7951 - val_recall: 0.7256 - val_auc: 0.8362 - 2s/epoch - 3ms/step
Epoch 20/50
749/749 - 2s - loss: 0.5258 - accuracy: 0.7485 - precision: 0.7722 - recall: 0.7448 - auc: 0.8272 - val_loss: 0.5206 - val_accuracy: 0.7526 - val_precision: 0.7620 - val_recall: 0.7746 - val_auc: 0.8337 - 2s/epoch - 3ms/step
Epoch 21/50
749/749 - 3s - loss: 0.5261 - accuracy: 0.7512 - precision: 0.7753 - recall: 0.7465 - auc: 0.8272 - val_loss: 0.5139 - val_accuracy: 0.7562 - val_precision: 0.7670 - val_recall: 0.7749 - val_auc: 0.8362 - 3s/epoch - 4ms/step
Epoch 22/50
749/749 - 3s - loss: 0.5242 - accuracy: 0.7519 - precision: 0.7740 - recall: 0.7505 - auc: 0.8283 - val_loss: 0.5231 - val_accuracy: 0.7512 - val_precision: 0.7786 - val_recall: 0.7408 - val_auc: 0.8312 - 3s/epoch - 4ms/step
Epoch 23/50
749/749 - 2s - loss: 0.5239 - accuracy: 0.7513 - precision: 0.7749 - recall: 0.7473 - auc: 0.8287 - val_loss: 0.5178 - val_accuracy: 0.7559 - val_precision: 0.7792 - val_recall: 0.7519 - val_auc: 0.8330 - 2s/epoch - 3ms/step
Epoch 24/50
749/749 - 2s - loss: 0.5236 - accuracy: 0.7507 - precision: 0.7739 - recall: 0.7475 - auc: 0.8289 - val_loss: 0.5167 - val_accuracy: 0.7555 - val_precision: 0.7619 - val_recall: 0.7827 - val_auc: 0.8343 - 2s/epoch - 3ms/step
Epoch 25/50
749/749 - 2s - loss: 0.5233 - accuracy: 0.7514 - precision: 0.7755 - recall: 0.7465 - auc: 0.8285 - val_loss: 0.5132 - val_accuracy: 0.7550 - val_precision: 0.7870 - val_recall: 0.7365 - val_auc: 0.8378 - 2s/epoch - 3ms/step
Epoch 26/50
749/749 - 2s - loss: 0.5216 - accuracy: 0.7503 - precision: 0.7731 - recall: 0.7480 - auc: 0.8299 - val_loss: 0.5093 - val_accuracy: 0.7623 - val_precision: 0.7640 - val_recall: 0.7971 - val_auc: 0.8399 - 2s/epoch - 3ms/step
Epoch 27/50
749/749 - 3s - loss: 0.5207 - accuracy: 0.7506 - precision: 0.7749 - recall: 0.7456 - auc: 0.8304 - val_loss: 0.5089 - val_accuracy: 0.7587 - val_precision: 0.7740 - val_recall: 0.7687 - val_auc: 0.8392 - 3s/epoch - 4ms/step
Epoch 28/50
749/749 - 3s - loss: 0.5214 - accuracy: 0.7525 - precision: 0.7760 - recall: 0.7487 - auc: 0.8306 - val_loss: 0.5114 - val_accuracy: 0.7587 - val_precision: 0.7695 - val_recall: 0.7770 - val_auc: 0.8384 - 3s/epoch - 4ms/step
Epoch 29/50
749/749 - 2s - loss: 0.5194 - accuracy: 0.7527 - precision: 0.7767 - recall: 0.7479 - auc: 0.8311 - val_loss: 0.5090 - val_accuracy: 0.7592 - val_precision: 0.7754 - val_recall: 0.7675 - val_auc: 0.8389 - 2s/epoch - 3ms/step
Epoch 30/50
749/749 - 2s - loss: 0.5194 - accuracy: 0.7529 - precision: 0.7756 - recall: 0.7502 - auc: 0.8311 - val_loss: 0.5124 - val_accuracy: 0.7571 - val_precision: 0.7788 - val_recall: 0.7559 - val_auc: 0.8380 - 2s/epoch - 3ms/step
Epoch 31/50
749/749 - 2s - loss: 0.5182 - accuracy: 0.7537 - precision: 0.7777 - recall: 0.7490 - auc: 0.8324 - val_loss: 0.5122 - val_accuracy: 0.7561 - val_precision: 0.7717 - val_recall: 0.7659 - val_auc: 0.8371 - 2s/epoch - 3ms/step
Epoch 32/50
749/749 - 2s - loss: 0.5180 - accuracy: 0.7532 - precision: 0.7771 - recall: 0.7486 - auc: 0.8324 - val_loss: 0.5135 - val_accuracy: 0.7570 - val_precision: 0.7688 - val_recall: 0.7737 - val_auc: 0.8367 - 2s/epoch - 3ms/step
Epoch 33/50
749/749 - 3s - loss: 0.5194 - accuracy: 0.7554 - precision: 0.7787 - recall: 0.7514 - auc: 0.8319 - val_loss: 0.5119 - val_accuracy: 0.7561 - val_precision: 0.7609 - val_recall: 0.7865 - val_auc: 0.8362 - 3s/epoch - 4ms/step
Epoch 34/50
749/749 - 3s - loss: 0.5181 - accuracy: 0.7552 - precision: 0.7774 - recall: 0.7534 - auc: 0.8326 - val_loss: 0.5244 - val_accuracy: 0.7532 - val_precision: 0.7839 - val_recall: 0.7370 - val_auc: 0.8299 - 3s/epoch - 4ms/step
Epoch 35/50
749/749 - 2s - loss: 0.5197 - accuracy: 0.7532 - precision: 0.7792 - recall: 0.7450 - auc: 0.8321 - val_loss: 0.5102 - val_accuracy: 0.7587 - val_precision: 0.7806 - val_recall: 0.7571 - val_auc: 0.8390 - 2s/epoch - 3ms/step
Epoch 36/50
749/749 - 2s - loss: 0.5173 - accuracy: 0.7547 - precision: 0.7793 - recall: 0.7488 - auc: 0.8325 - val_loss: 0.5108 - val_accuracy: 0.7607 - val_precision: 0.7605 - val_recall: 0.8000 - val_auc: 0.8384 - 2s/epoch - 3ms/step
Epoch 37/50
749/749 - 2s - loss: 0.5169 - accuracy: 0.7540 - precision: 0.7779 - recall: 0.7494 - auc: 0.8327 - val_loss: 0.5216 - val_accuracy: 0.7511 - val_precision: 0.7827 - val_recall: 0.7334 - val_auc: 0.8333 - 2s/epoch - 3ms/step
Epoch 38/50
749/749 - 2s - loss: 0.5179 - accuracy: 0.7541 - precision: 0.7776 - recall: 0.7501 - auc: 0.8323 - val_loss: 0.5064 - val_accuracy: 0.7624 - val_precision: 0.7749 - val_recall: 0.7768 - val_auc: 0.8404 - 2s/epoch - 3ms/step
Epoch 39/50
749/749 - 3s - loss: 0.5167 - accuracy: 0.7553 - precision: 0.7783 - recall: 0.7521 - auc: 0.8330 - val_loss: 0.5081 - val_accuracy: 0.7601 - val_precision: 0.7741 - val_recall: 0.7723 - val_auc: 0.8391 - 3s/epoch - 4ms/step
Epoch 40/50
749/749 - 3s - loss: 0.5177 - accuracy: 0.7542 - precision: 0.7797 - recall: 0.7467 - auc: 0.8326 - val_loss: 0.5123 - val_accuracy: 0.7561 - val_precision: 0.7568 - val_recall: 0.7947 - val_auc: 0.8360 - 3s/epoch - 4ms/step
Epoch 41/50
749/749 - 2s - loss: 0.5163 - accuracy: 0.7548 - precision: 0.7785 - recall: 0.7503 - auc: 0.8334 - val_loss: 0.5106 - val_accuracy: 0.7569 - val_precision: 0.7891 - val_recall: 0.7379 - val_auc: 0.8402 - 2s/epoch - 3ms/step
Epoch 42/50
749/749 - 2s - loss: 0.5184 - accuracy: 0.7551 - precision: 0.7798 - recall: 0.7488 - auc: 0.8329 - val_loss: 0.5075 - val_accuracy: 0.7615 - val_precision: 0.7901 - val_recall: 0.7483 - val_auc: 0.8412 - 2s/epoch - 3ms/step
Epoch 43/50
749/749 - 2s - loss: 0.5158 - accuracy: 0.7549 - precision: 0.7787 - recall: 0.7502 - auc: 0.8337 - val_loss: 0.5135 - val_accuracy: 0.7590 - val_precision: 0.7988 - val_recall: 0.7282 - val_auc: 0.8402 - 2s/epoch - 3ms/step
Epoch 44/50
749/749 - 2s - loss: 0.5158 - accuracy: 0.7538 - precision: 0.7794 - recall: 0.7462 - auc: 0.8340 - val_loss: 0.5074 - val_accuracy: 0.7624 - val_precision: 0.7898 - val_recall: 0.7509 - val_auc: 0.8415 - 2s/epoch - 3ms/step
Epoch 45/50
749/749 - 5s - loss: 0.5155 - accuracy: 0.7554 - precision: 0.7799 - recall: 0.7495 - auc: 0.8341 - val_loss: 0.5051 - val_accuracy: 0.7658 - val_precision: 0.7775 - val_recall: 0.7810 - val_auc: 0.8419 - 5s/epoch - 6ms/step
Epoch 46/50
749/749 - 5s - loss: 0.5157 - accuracy: 0.7571 - precision: 0.7809 - recall: 0.7524 - auc: 0.8345 - val_loss: 0.5119 - val_accuracy: 0.7594 - val_precision: 0.7989 - val_recall: 0.7289 - val_auc: 0.8403 - 5s/epoch - 6ms/step
Epoch 47/50
749/749 - 5s - loss: 0.5158 - accuracy: 0.7554 - precision: 0.7811 - recall: 0.7473 - auc: 0.8341 - val_loss: 0.5075 - val_accuracy: 0.7649 - val_precision: 0.7793 - val_recall: 0.7756 - val_auc: 0.8408 - 5s/epoch - 6ms/step
Epoch 48/50
749/749 - 6s - loss: 0.5161 - accuracy: 0.7541 - precision: 0.7780 - recall: 0.7495 - auc: 0.8335 - val_loss: 0.5089 - val_accuracy: 0.7604 - val_precision: 0.7906 - val_recall: 0.7446 - val_auc: 0.8404 - 6s/epoch - 9ms/step
Epoch 49/50
749/749 - 5s - loss: 0.5146 - accuracy: 0.7568 - precision: 0.7813 - recall: 0.7508 - auc: 0.8344 - val_loss: 0.5051 - val_accuracy: 0.7626 - val_precision: 0.7882 - val_recall: 0.7543 - val_auc: 0.8419 - 5s/epoch - 6ms/step
Epoch 50/50
749/749 - 2s - loss: 0.5158 - accuracy: 0.7554 - precision: 0.7796 - recall: 0.7499 - auc: 0.8340 - val_loss: 0.5077 - val_accuracy: 0.7633 - val_precision: 0.7865 - val_recall: 0.7588 - val_auc: 0.8401 - 2s/epoch - 3ms/step

Resultados

plot_metrics(history)

train_pred = model.predict(X_train)
test_pred = model.predict(X_test)

print('Número de exemplos positivos do conjunto de teste =', len(y_test[y_test > 0.9]))
results = model.evaluate(X_test, y_test, verbose=0)

for name, value in zip(model.metrics_names, results):
    print(f'{name}: {value:.4f}')

749/749 [==============================] - 2s 2ms/step
250/250 [==============================] - 0s 2ms/step
Número de exemplos positivos do conjunto de teste = 4224
loss: 0.5077
accuracy: 0.7633
precision: 0.7865
recall: 0.7588
auc: 0.8401

precision = results[2]
recall = results[3]
F1 = 2*precision*recall/(precision + recall)
print(f'Pontuação F1 = {F1:.4f}')

Pontuação F1 = 0.7724

ConfusionMatrixDisplay.from_predictions(y_test, np.round(test_pred),
                                        normalize='true', cmap='Blues',
                                        values_format='.3f')
plt.show()

from sklearn.metrics import balanced_accuracy_score

balanced_accuracy_score(y_test, np.round(test_pred))

0.7635341955675261

fp_train, tp_train, _ = roc_curve(y_train, train_pred)
fp_test, tp_test, _ = roc_curve(y_test, test_pred)
plt.figure(figsize=(8, 6))
plt.plot(100*fp_train, 100*tp_train, 'b', label='Dados treinamento')
plt.plot(100*fp_test, 100*tp_test, 'r', label='Dados teste')
plt.xlabel('Positivos falsos [%]')
plt.ylabel('Positivos verdadeiros [%]')
plt.xlim([0,100])
plt.ylim([0,100])
plt.grid(True)
plt.legend()
plt.show()

custo_e_metricas_train = model.evaluate(X_train, y_train)
# custo_e_metricas_val = rna_reg.evaluate(X_val_norm, y_val)
custo_e_metricas_test = model.evaluate(X_test, y_test)

749/749 [==============================] - 1s 2ms/step - loss: 0.5117 - accuracy: 0.7564 - precision: 0.7817 - recall: 0.7492 - auc: 0.8366
250/250 [==============================] - 0s 2ms/step - loss: 0.5077 - accuracy: 0.7633 - precision: 0.7865 - recall: 0.7588 - auc: 0.8401

Criação e treinamento da RNA Residual

# Função que cria o bloco para a RN residual
def bloco_residual(x, n):

    z1 = Dense(n, activation='relu')(x)
    z2 = Dense(n)(z1)
    sum = Add()([x, z2])
    a2 = Activation('relu')(sum)

    return a2

# Definição da entrada
input_shape = X_train.shape[1:]
input_features = Input(shape=input_shape)

X1 = Dense(64, activation='relu')(input_features)

X2 = bloco_residual(X1, 64)

X3 = Dense(64, activation='relu')(X2)

X4 = bloco_residual(X3, 64)

X5 = Dense(16, activation='relu')(X4)

Y = Dense(units=1, activation='sigmoid')(X5)

# Criação da RNA
rna = Model(input_features, Y)

# Mostra resumo da RNA
rna.summary()

Model: "model_1"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to
==================================================================================================
 input_1 (InputLayer)           [(None, 25)]         0           []

 dense_4 (Dense)                (None, 64)           1664        ['input_1[0][0]']

 dense_5 (Dense)                (None, 64)           4160        ['dense_4[0][0]']

 dense_6 (Dense)                (None, 64)           4160        ['dense_5[0][0]']

 add (Add)                      (None, 64)           0           ['dense_4[0][0]',
                                                                  'dense_6[0][0]']

 activation (Activation)        (None, 64)           0           ['add[0][0]']

 dense_7 (Dense)                (None, 64)           4160        ['activation[0][0]']

 dense_8 (Dense)                (None, 64)           4160        ['dense_7[0][0]']

 dense_9 (Dense)                (None, 64)           4160        ['dense_8[0][0]']

 add_1 (Add)                    (None, 64)           0           ['dense_7[0][0]',
                                                                  'dense_9[0][0]']

 activation_1 (Activation)      (None, 64)           0           ['add_1[0][0]']

 dense_10 (Dense)               (None, 16)           1040        ['activation_1[0][0]']

 dense_11 (Dense)               (None, 1)            17          ['dense_10[0][0]']

==================================================================================================
Total params: 23,521
Trainable params: 23,521
Non-trainable params: 0
__________________________________________________________________________________________________

keras.utils.plot_model(rna, show_shapes=True)

from tensorflow.keras.callbacks import EarlyStopping

# Define métricas
METRICS = [keras.metrics.BinaryAccuracy(name='accuracy'),
           keras.metrics.Precision(name='precision'),
           keras.metrics.Recall(name='recall'),
           keras.metrics.AUC(name='auc')]

call_es = EarlyStopping(monitor='val_loss', patience=20,
                        restore_best_weights=True)

adam = Adam(learning_rate=0.001)
rna.compile(optimizer=adam, loss='binary_crossentropy',
              metrics=METRICS)

history = rna.fit(X_train, y_train, epochs=50,
                    class_weight=class_weight,
                    verbose=2, batch_size=32,
                    validation_data=(X_test, y_test),
                    callbacks=[call_es])

Epoch 1/50
749/749 - 5s - loss: 0.5518 - accuracy: 0.7238 - precision: 0.7489 - recall: 0.7196 - auc: 0.7946 - val_loss: 0.5110 - val_accuracy: 0.7500 - val_precision: 0.7364 - val_recall: 0.8220 - val_auc: 0.8298 - 5s/epoch - 6ms/step
Epoch 2/50
749/749 - 4s - loss: 0.5166 - accuracy: 0.7464 - precision: 0.7704 - recall: 0.7421 - auc: 0.8235 - val_loss: 0.5102 - val_accuracy: 0.7468 - val_precision: 0.8176 - val_recall: 0.6716 - val_auc: 0.8377 - 4s/epoch - 5ms/step
Epoch 3/50
749/749 - 2s - loss: 0.5051 - accuracy: 0.7549 - precision: 0.7826 - recall: 0.7436 - auc: 0.8320 - val_loss: 0.4951 - val_accuracy: 0.7572 - val_precision: 0.7858 - val_recall: 0.7443 - val_auc: 0.8404 - 2s/epoch - 3ms/step
Epoch 4/50
749/749 - 2s - loss: 0.4983 - accuracy: 0.7574 - precision: 0.7899 - recall: 0.7383 - auc: 0.8370 - val_loss: 0.4935 - val_accuracy: 0.7617 - val_precision: 0.7896 - val_recall: 0.7498 - val_auc: 0.8407 - 2s/epoch - 3ms/step
Epoch 5/50
749/749 - 2s - loss: 0.4912 - accuracy: 0.7612 - precision: 0.7924 - recall: 0.7439 - auc: 0.8422 - val_loss: 0.4948 - val_accuracy: 0.7581 - val_precision: 0.7976 - val_recall: 0.7277 - val_auc: 0.8412 - 2s/epoch - 3ms/step
Epoch 6/50
749/749 - 2s - loss: 0.4859 - accuracy: 0.7639 - precision: 0.7968 - recall: 0.7437 - auc: 0.8457 - val_loss: 0.4937 - val_accuracy: 0.7610 - val_precision: 0.7878 - val_recall: 0.7507 - val_auc: 0.8406 - 2s/epoch - 3ms/step
Epoch 7/50
749/749 - 3s - loss: 0.4821 - accuracy: 0.7655 - precision: 0.7998 - recall: 0.7431 - auc: 0.8483 - val_loss: 0.4924 - val_accuracy: 0.7626 - val_precision: 0.7815 - val_recall: 0.7656 - val_auc: 0.8420 - 3s/epoch - 4ms/step
Epoch 8/50
749/749 - 3s - loss: 0.4776 - accuracy: 0.7684 - precision: 0.8011 - recall: 0.7483 - auc: 0.8513 - val_loss: 0.5026 - val_accuracy: 0.7576 - val_precision: 0.8101 - val_recall: 0.7081 - val_auc: 0.8419 - 3s/epoch - 4ms/step
Epoch 9/50
749/749 - 2s - loss: 0.4747 - accuracy: 0.7699 - precision: 0.8059 - recall: 0.7448 - auc: 0.8537 - val_loss: 0.4993 - val_accuracy: 0.7591 - val_precision: 0.7957 - val_recall: 0.7332 - val_auc: 0.8375 - 2s/epoch - 3ms/step
Epoch 10/50
749/749 - 2s - loss: 0.4672 - accuracy: 0.7745 - precision: 0.8088 - recall: 0.7518 - auc: 0.8582 - val_loss: 0.5007 - val_accuracy: 0.7597 - val_precision: 0.7948 - val_recall: 0.7363 - val_auc: 0.8377 - 2s/epoch - 3ms/step
Epoch 11/50
749/749 - 2s - loss: 0.4636 - accuracy: 0.7758 - precision: 0.8085 - recall: 0.7555 - auc: 0.8607 - val_loss: 0.5073 - val_accuracy: 0.7566 - val_precision: 0.7857 - val_recall: 0.7429 - val_auc: 0.8350 - 2s/epoch - 3ms/step
Epoch 12/50
749/749 - 2s - loss: 0.4585 - accuracy: 0.7801 - precision: 0.8138 - recall: 0.7581 - auc: 0.8641 - val_loss: 0.5086 - val_accuracy: 0.7551 - val_precision: 0.7890 - val_recall: 0.7337 - val_auc: 0.8346 - 2s/epoch - 3ms/step
Epoch 13/50
749/749 - 4s - loss: 0.4527 - accuracy: 0.7840 - precision: 0.8183 - recall: 0.7610 - auc: 0.8678 - val_loss: 0.5106 - val_accuracy: 0.7619 - val_precision: 0.7785 - val_recall: 0.7689 - val_auc: 0.8329 - 4s/epoch - 5ms/step
Epoch 14/50
749/749 - 3s - loss: 0.4483 - accuracy: 0.7871 - precision: 0.8227 - recall: 0.7620 - auc: 0.8705 - val_loss: 0.5221 - val_accuracy: 0.7496 - val_precision: 0.7986 - val_recall: 0.7048 - val_auc: 0.8310 - 3s/epoch - 3ms/step
Epoch 15/50
749/749 - 2s - loss: 0.4418 - accuracy: 0.7882 - precision: 0.8238 - recall: 0.7633 - auc: 0.8743 - val_loss: 0.5200 - val_accuracy: 0.7506 - val_precision: 0.7744 - val_recall: 0.7462 - val_auc: 0.8254 - 2s/epoch - 3ms/step
Epoch 16/50
749/749 - 2s - loss: 0.4359 - accuracy: 0.7955 - precision: 0.8301 - recall: 0.7718 - auc: 0.8785 - val_loss: 0.5270 - val_accuracy: 0.7501 - val_precision: 0.7745 - val_recall: 0.7448 - val_auc: 0.8279 - 2s/epoch - 3ms/step
Epoch 17/50
749/749 - 2s - loss: 0.4280 - accuracy: 0.7981 - precision: 0.8316 - recall: 0.7756 - auc: 0.8831 - val_loss: 0.5304 - val_accuracy: 0.7574 - val_precision: 0.7819 - val_recall: 0.7512 - val_auc: 0.8302 - 2s/epoch - 3ms/step
Epoch 18/50
749/749 - 3s - loss: 0.4210 - accuracy: 0.8020 - precision: 0.8345 - recall: 0.7808 - auc: 0.8872 - val_loss: 0.5472 - val_accuracy: 0.7468 - val_precision: 0.7709 - val_recall: 0.7424 - val_auc: 0.8201 - 3s/epoch - 4ms/step
Epoch 19/50
749/749 - 3s - loss: 0.4132 - accuracy: 0.8078 - precision: 0.8396 - recall: 0.7873 - auc: 0.8917 - val_loss: 0.5427 - val_accuracy: 0.7498 - val_precision: 0.7798 - val_recall: 0.7351 - val_auc: 0.8241 - 3s/epoch - 4ms/step
Epoch 20/50
749/749 - 2s - loss: 0.4067 - accuracy: 0.8098 - precision: 0.8411 - recall: 0.7900 - auc: 0.8947 - val_loss: 0.5607 - val_accuracy: 0.7458 - val_precision: 0.7827 - val_recall: 0.7197 - val_auc: 0.8184 - 2s/epoch - 3ms/step
Epoch 21/50
749/749 - 2s - loss: 0.3976 - accuracy: 0.8153 - precision: 0.8477 - recall: 0.7937 - auc: 0.8999 - val_loss: 0.5638 - val_accuracy: 0.7368 - val_precision: 0.7541 - val_recall: 0.7462 - val_auc: 0.8132 - 2s/epoch - 3ms/step
Epoch 22/50
749/749 - 2s - loss: 0.3921 - accuracy: 0.8176 - precision: 0.8502 - recall: 0.7958 - auc: 0.9030 - val_loss: 0.5656 - val_accuracy: 0.7368 - val_precision: 0.7695 - val_recall: 0.7178 - val_auc: 0.8100 - 2s/epoch - 3ms/step
Epoch 23/50
749/749 - 2s - loss: 0.3824 - accuracy: 0.8235 - precision: 0.8553 - recall: 0.8023 - auc: 0.9078 - val_loss: 0.5929 - val_accuracy: 0.7381 - val_precision: 0.7721 - val_recall: 0.7169 - val_auc: 0.8075 - 2s/epoch - 3ms/step
Epoch 24/50
749/749 - 4s - loss: 0.3792 - accuracy: 0.8264 - precision: 0.8593 - recall: 0.8037 - auc: 0.9094 - val_loss: 0.6017 - val_accuracy: 0.7323 - val_precision: 0.7592 - val_recall: 0.7240 - val_auc: 0.8014 - 4s/epoch - 5ms/step
Epoch 25/50
749/749 - 3s - loss: 0.3683 - accuracy: 0.8308 - precision: 0.8634 - recall: 0.8083 - auc: 0.9149 - val_loss: 0.6144 - val_accuracy: 0.7388 - val_precision: 0.7567 - val_recall: 0.7467 - val_auc: 0.8101 - 3s/epoch - 3ms/step
Epoch 26/50
749/749 - 2s - loss: 0.3630 - accuracy: 0.8337 - precision: 0.8622 - recall: 0.8164 - auc: 0.9172 - val_loss: 0.6191 - val_accuracy: 0.7320 - val_precision: 0.7589 - val_recall: 0.7237 - val_auc: 0.8027 - 2s/epoch - 3ms/step
Epoch 27/50
749/749 - 2s - loss: 0.3540 - accuracy: 0.8391 - precision: 0.8700 - recall: 0.8184 - auc: 0.9213 - val_loss: 0.6326 - val_accuracy: 0.7366 - val_precision: 0.7603 - val_recall: 0.7337 - val_auc: 0.8029 - 2s/epoch - 3ms/step

Resultados

plot_metrics(history)

train_pred = rna.predict(X_train)
test_pred = rna.predict(X_test)

print('Número de exemplos positivos do conjunto de teste =', len(y_test[y_test > 0.9]))
results = rna.evaluate(X_test, y_test, verbose=0)

for name, value in zip(rna.metrics_names, results):
    print(f'{name}: {value:.4f}')

749/749 [==============================] - 1s 1ms/step
250/250 [==============================] - 0s 2ms/step
Número de exemplos positivos do conjunto de teste = 4224
loss: 0.4924
accuracy: 0.7626
precision: 0.7815
recall: 0.7656
auc: 0.8420

precision = results[2]
recall = results[3]
F1 = 2*precision*recall/(precision + recall)
print(f'Pontuação F1 = {F1:.4f}')

Pontuação F1 = 0.7735

ConfusionMatrixDisplay.from_predictions(y_test, np.round(test_pred),
                                        normalize='true', cmap='Blues',
                                        values_format='.3f')
plt.show()

from sklearn.metrics import balanced_accuracy_score

balanced_accuracy_score(y_test, np.round(test_pred))

0.7624396637816244

fp_train, tp_train, _ = roc_curve(y_train, train_pred)
fp_test, tp_test, _ = roc_curve(y_test, test_pred)
plt.figure(figsize=(8, 6))
plt.plot(100*fp_train, 100*tp_train, 'b', label='Dados treinamento')
plt.plot(100*fp_test, 100*tp_test, 'r', label='Dados teste')
plt.xlabel('Positivos falsos [%]')
plt.ylabel('Positivos verdadeiros [%]')
plt.xlim([0,100])
plt.ylim([0,100])
plt.grid(True)
plt.legend()
plt.show()

custo_e_metricas_train = rna.evaluate(X_train, y_train)
custo_e_metricas_test = rna.evaluate(X_test, y_test)

749/749 [==============================] - 2s 3ms/step - loss: 0.4663 - accuracy: 0.7756 - precision: 0.7976 - recall: 0.7722 - auc: 0.8589
250/250 [==============================] - 0s 2ms/step - loss: 0.4924 - accuracy: 0.7626 - precision: 0.7815 - recall: 0.7656 - auc: 0.8420

Criação e treinamento da RNA Sequencial

neg, pos = np.bincount(y_train)
total = neg + pos
print(f'Exemplos:\n Total: {total}\n Positivos: {pos} ({100*pos/total:.2f}% do total)')

# Cálculo dos pesos das duas classe
weight_for_0 = (1 / neg)*(total)/2.0
weight_for_1 = (1 / pos)*(total)/2.0

# Dicionário de pesos das classes para treinamento
class_weight = {0: weight_for_0, 1: weight_for_1}
print('Peso da classe 0: {:.2f}'.format(weight_for_0))
print('Peso da classe 1: {:.2f}'.format(weight_for_1))

Exemplos:
 Total: 23937
 Positivos: 12673 (52.94% do total)
Peso da classe 0: 1.06
Peso da classe 1: 0.94

rna = Sequential()
rna.add(Dense(units=128, activation='relu', input_shape=X_train.shape[1:]))
rna.add(Dense(units=128, activation='relu'))
rna.add(Dense(units=32, activation='relu'))
rna.add(Dense(units=1, activation='sigmoid'))
rna.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 dense_12 (Dense)            (None, 128)               3328

 dense_13 (Dense)            (None, 128)               16512

 dense_14 (Dense)            (None, 32)                4128

 dense_15 (Dense)            (None, 1)                 33

=================================================================
Total params: 24,001
Trainable params: 24,001
Non-trainable params: 0
_________________________________________________________________

keras.utils.plot_model(rna, show_shapes=True)

from tensorflow.keras.callbacks import EarlyStopping

# Define métricas
METRICS = [keras.metrics.BinaryAccuracy(name='accuracy'),
           keras.metrics.Precision(name='precision'),
           keras.metrics.Recall(name='recall'),
           keras.metrics.AUC(name='auc')]

call_es = EarlyStopping(monitor='val_loss', patience=20,
                        restore_best_weights=True)

adam = Adam(learning_rate=0.001)
rna.compile(optimizer=adam, loss='binary_crossentropy',
            metrics=METRICS)

history = rna.fit(X_train, y_train, epochs=50,
                  class_weight=class_weight,
                  verbose=2, batch_size=32,
                  validation_data=(X_test, y_test),
                  callbacks=[call_es])

Epoch 1/50
749/749 - 4s - loss: 0.5358 - accuracy: 0.7334 - precision: 0.7567 - recall: 0.7316 - auc: 0.8085 - val_loss: 0.5022 - val_accuracy: 0.7604 - val_precision: 0.7605 - val_recall: 0.7990 - val_auc: 0.8361 - 4s/epoch - 5ms/step
Epoch 2/50
749/749 - 2s - loss: 0.5106 - accuracy: 0.7494 - precision: 0.7735 - recall: 0.7449 - auc: 0.8279 - val_loss: 0.4973 - val_accuracy: 0.7580 - val_precision: 0.7962 - val_recall: 0.7296 - val_auc: 0.8401 - 2s/epoch - 3ms/step
Epoch 3/50
749/749 - 2s - loss: 0.5016 - accuracy: 0.7560 - precision: 0.7844 - recall: 0.7435 - auc: 0.8348 - val_loss: 0.4937 - val_accuracy: 0.7595 - val_precision: 0.7793 - val_recall: 0.7614 - val_auc: 0.8399 - 2s/epoch - 3ms/step
Epoch 4/50
749/749 - 2s - loss: 0.4953 - accuracy: 0.7574 - precision: 0.7859 - recall: 0.7447 - auc: 0.8393 - val_loss: 0.4940 - val_accuracy: 0.7597 - val_precision: 0.7998 - val_recall: 0.7285 - val_auc: 0.8422 - 2s/epoch - 3ms/step
Epoch 5/50
749/749 - 3s - loss: 0.4892 - accuracy: 0.7615 - precision: 0.7921 - recall: 0.7450 - auc: 0.8433 - val_loss: 0.4907 - val_accuracy: 0.7645 - val_precision: 0.7826 - val_recall: 0.7687 - val_auc: 0.8427 - 3s/epoch - 4ms/step
Epoch 6/50
749/749 - 2s - loss: 0.4854 - accuracy: 0.7643 - precision: 0.7932 - recall: 0.7503 - auc: 0.8461 - val_loss: 0.5065 - val_accuracy: 0.7544 - val_precision: 0.8227 - val_recall: 0.6832 - val_auc: 0.8422 - 2s/epoch - 3ms/step
Epoch 7/50
749/749 - 2s - loss: 0.4831 - accuracy: 0.7674 - precision: 0.7975 - recall: 0.7515 - auc: 0.8479 - val_loss: 0.4959 - val_accuracy: 0.7612 - val_precision: 0.8113 - val_recall: 0.7154 - val_auc: 0.8431 - 2s/epoch - 3ms/step
Epoch 8/50
749/749 - 2s - loss: 0.4758 - accuracy: 0.7693 - precision: 0.8033 - recall: 0.7472 - auc: 0.8524 - val_loss: 0.4903 - val_accuracy: 0.7612 - val_precision: 0.7918 - val_recall: 0.7448 - val_auc: 0.8435 - 2s/epoch - 3ms/step
Epoch 9/50
749/749 - 2s - loss: 0.4708 - accuracy: 0.7751 - precision: 0.8072 - recall: 0.7556 - auc: 0.8558 - val_loss: 0.4937 - val_accuracy: 0.7645 - val_precision: 0.7956 - val_recall: 0.7472 - val_auc: 0.8415 - 2s/epoch - 3ms/step
Epoch 10/50
749/749 - 2s - loss: 0.4659 - accuracy: 0.7767 - precision: 0.8094 - recall: 0.7565 - auc: 0.8595 - val_loss: 0.4994 - val_accuracy: 0.7621 - val_precision: 0.7642 - val_recall: 0.7964 - val_auc: 0.8395 - 2s/epoch - 3ms/step
Epoch 11/50
749/749 - 3s - loss: 0.4627 - accuracy: 0.7793 - precision: 0.8103 - recall: 0.7614 - auc: 0.8615 - val_loss: 0.4941 - val_accuracy: 0.7616 - val_precision: 0.7854 - val_recall: 0.7564 - val_auc: 0.8413 - 3s/epoch - 4ms/step
Epoch 12/50
749/749 - 3s - loss: 0.4562 - accuracy: 0.7824 - precision: 0.8136 - recall: 0.7641 - auc: 0.8654 - val_loss: 0.5038 - val_accuracy: 0.7531 - val_precision: 0.8036 - val_recall: 0.7062 - val_auc: 0.8383 - 3s/epoch - 3ms/step
Epoch 13/50
749/749 - 2s - loss: 0.4508 - accuracy: 0.7844 - precision: 0.8146 - recall: 0.7675 - auc: 0.8691 - val_loss: 0.5031 - val_accuracy: 0.7594 - val_precision: 0.7639 - val_recall: 0.7895 - val_auc: 0.8365 - 2s/epoch - 3ms/step
Epoch 14/50
749/749 - 2s - loss: 0.4457 - accuracy: 0.7884 - precision: 0.8184 - recall: 0.7714 - auc: 0.8720 - val_loss: 0.5099 - val_accuracy: 0.7572 - val_precision: 0.8112 - val_recall: 0.7057 - val_auc: 0.8397 - 2s/epoch - 3ms/step
Epoch 15/50
749/749 - 2s - loss: 0.4397 - accuracy: 0.7914 - precision: 0.8226 - recall: 0.7727 - auc: 0.8758 - val_loss: 0.5073 - val_accuracy: 0.7592 - val_precision: 0.7770 - val_recall: 0.7647 - val_auc: 0.8369 - 2s/epoch - 3ms/step
Epoch 16/50
749/749 - 2s - loss: 0.4338 - accuracy: 0.7949 - precision: 0.8250 - recall: 0.7776 - auc: 0.8795 - val_loss: 0.5237 - val_accuracy: 0.7554 - val_precision: 0.8028 - val_recall: 0.7131 - val_auc: 0.8353 - 2s/epoch - 3ms/step
Epoch 17/50
749/749 - 3s - loss: 0.4286 - accuracy: 0.7974 - precision: 0.8258 - recall: 0.7824 - auc: 0.8826 - val_loss: 0.5227 - val_accuracy: 0.7566 - val_precision: 0.7926 - val_recall: 0.7318 - val_auc: 0.8357 - 3s/epoch - 4ms/step
Epoch 18/50
749/749 - 3s - loss: 0.4235 - accuracy: 0.7997 - precision: 0.8306 - recall: 0.7810 - auc: 0.8855 - val_loss: 0.5222 - val_accuracy: 0.7532 - val_precision: 0.7732 - val_recall: 0.7554 - val_auc: 0.8308 - 3s/epoch - 4ms/step
Epoch 19/50
749/749 - 2s - loss: 0.4148 - accuracy: 0.8073 - precision: 0.8359 - recall: 0.7914 - auc: 0.8903 - val_loss: 0.5332 - val_accuracy: 0.7496 - val_precision: 0.7914 - val_recall: 0.7157 - val_auc: 0.8267 - 2s/epoch - 3ms/step
Epoch 20/50
749/749 - 2s - loss: 0.4109 - accuracy: 0.8069 - precision: 0.8370 - recall: 0.7888 - auc: 0.8927 - val_loss: 0.5356 - val_accuracy: 0.7523 - val_precision: 0.7814 - val_recall: 0.7389 - val_auc: 0.8301 - 2s/epoch - 3ms/step
Epoch 21/50
749/749 - 2s - loss: 0.4016 - accuracy: 0.8140 - precision: 0.8394 - recall: 0.8020 - auc: 0.8978 - val_loss: 0.5497 - val_accuracy: 0.7448 - val_precision: 0.7613 - val_recall: 0.7545 - val_auc: 0.8190 - 2s/epoch - 3ms/step
Epoch 22/50
749/749 - 2s - loss: 0.3946 - accuracy: 0.8147 - precision: 0.8422 - recall: 0.7999 - auc: 0.9015 - val_loss: 0.5507 - val_accuracy: 0.7491 - val_precision: 0.7634 - val_recall: 0.7623 - val_auc: 0.8172 - 2s/epoch - 3ms/step
Epoch 23/50
749/749 - 3s - loss: 0.3885 - accuracy: 0.8203 - precision: 0.8465 - recall: 0.8069 - auc: 0.9046 - val_loss: 0.5611 - val_accuracy: 0.7452 - val_precision: 0.7632 - val_recall: 0.7521 - val_auc: 0.8187 - 3s/epoch - 4ms/step
Epoch 24/50
749/749 - 3s - loss: 0.3811 - accuracy: 0.8240 - precision: 0.8535 - recall: 0.8060 - auc: 0.9084 - val_loss: 0.5730 - val_accuracy: 0.7359 - val_precision: 0.7877 - val_recall: 0.6861 - val_auc: 0.8117 - 3s/epoch - 4ms/step
Epoch 25/50
749/749 - 2s - loss: 0.3751 - accuracy: 0.8269 - precision: 0.8553 - recall: 0.8101 - auc: 0.9112 - val_loss: 0.5810 - val_accuracy: 0.7404 - val_precision: 0.7471 - val_recall: 0.7706 - val_auc: 0.8102 - 2s/epoch - 3ms/step
Epoch 26/50
749/749 - 2s - loss: 0.3678 - accuracy: 0.8304 - precision: 0.8562 - recall: 0.8168 - auc: 0.9152 - val_loss: 0.5891 - val_accuracy: 0.7411 - val_precision: 0.7795 - val_recall: 0.7124 - val_auc: 0.8153 - 2s/epoch - 3ms/step
Epoch 27/50
749/749 - 2s - loss: 0.3616 - accuracy: 0.8322 - precision: 0.8580 - recall: 0.8186 - auc: 0.9176 - val_loss: 0.6028 - val_accuracy: 0.7411 - val_precision: 0.7782 - val_recall: 0.7145 - val_auc: 0.8141 - 2s/epoch - 3ms/step
Epoch 28/50
749/749 - 2s - loss: 0.3541 - accuracy: 0.8377 - precision: 0.8633 - recall: 0.8240 - auc: 0.9216 - val_loss: 0.6265 - val_accuracy: 0.7387 - val_precision: 0.7814 - val_recall: 0.7031 - val_auc: 0.8125 - 2s/epoch - 2ms/step

Resultados

plot_metrics(history)

train_pred = rna.predict(X_train)
test_pred = rna.predict(X_test)

print('Número de exemplos positivos do conjunto de teste =', len(y_test[y_test > 0.9]))
results = rna.evaluate(X_test, y_test, verbose=0)

for name, value in zip(rna.metrics_names, results):
    print(f'{name}: {value:.4f}')

749/749 [==============================] - 1s 1ms/step
250/250 [==============================] - 0s 1ms/step
Número de exemplos positivos do conjunto de teste = 4224
loss: 0.4903
accuracy: 0.7612
precision: 0.7918
recall: 0.7448
auc: 0.8435

precision = results[2]
recall = results[3]
F1 = 2*precision*recall/(precision + recall)
print(f'Pontuação F1 = {F1:.4f}')

Pontuação F1 = 0.7676

ConfusionMatrixDisplay.from_predictions(y_test, np.round(test_pred),
                                        normalize='true', cmap='Blues',
                                        values_format='.3f')
plt.show()

from sklearn.metrics import balanced_accuracy_score

balanced_accuracy_score(y_test, np.round(test_pred))

0.762275993120284

fp_train, tp_train, _ = roc_curve(y_train, train_pred)
fp_test, tp_test, _ = roc_curve(y_test, test_pred)
plt.figure(figsize=(8, 6))
plt.plot(100*fp_train, 100*tp_train, 'b', label='Dados treinamento')
plt.plot(100*fp_test, 100*tp_test, 'r', label='Dados teste')
plt.xlabel('Positivos falsos [%]')
plt.ylabel('Positivos verdadeiros [%]')
plt.xlim([0,100])
plt.ylim([0,100])
plt.grid(True)
plt.legend()
plt.show()

custo_e_metricas_train = rna.evaluate(X_train, y_train)
custo_e_metricas_test = rna.evaluate(X_test, y_test)

749/749 [==============================] - 1s 2ms/step - loss: 0.4608 - accuracy: 0.7789 - precision: 0.8133 - recall: 0.7558 - auc: 0.8638
250/250 [==============================] - 0s 2ms/step - loss: 0.4903 - accuracy: 0.7612 - precision: 0.7918 - recall: 0.7448 - auc: 0.8435